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ABSTRACT 

An  overview  of  post  World  War  II  U.S.  naval  ship  acqui- 
sition policies  and  results  is  presented.   Ihe  major  emphasis 
in  the  overview  is  on  s eve  ml  new  control  systems  and  policies 
announced  since  1970. 

A  case  for  development  cf  a  comprehensive,  life-cycle 
ship  success  evaluation  system  is  based  on  the  current  domi- 
nant emphasis  on  short-term  goal  a'^hievement  and  the  need  for 
a  centralized,  complete  historical  record  and  data  bank  on 
ship  acquisitions. 

Success  in  naval  f^hip  acquisitions  is  defined  as  hpw 
well  actual  results  match  a  set  of  prioritized  goals  determ- 
ined from  the  requirements  of  the  military  need.   Individual 
sets  of  criteria  for  performance,  cost  and  schedule  goals  are 
developed.   Two  schemes  for  evaluating  overall  ship 


acqtilsition  success,  a  "manual"  model  and  a  computerized 
simulation  model,  are  outlined. 

The  development  of  a  comprehensive  life-cycle  ship 
acquisition  evaluation  system  is  recommended.   Further 
study  rnd  development  of  certain  aspects  of  the  proposed 
system  are  required. 
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INTRODUCTION 

The  following  excerpts  are  taken  from  an  article  in 
the  March  4,  1976,  edition  of  the  Boston  Globe. 

"  a  shipbuilder  (is)  seeking  payment  of 

$11,708  owed  by  the  Federal  Government 

for  a  Navy  (ship)  the  300  foot  (ship) 

cost  the  Navy  $75,000,  plus  a  $16,000 

overrun  resulting  from  requested  modifications 

and  delays  in  subcontract  work,  the 

unpaid  debt  was  a  major  reason  why  (the  ship- 
yard) ultimately  went  bankrupt  " 

Only  the  magnitude  of  the  figures  set  this  article  apart  from 
similar,  almost  daily  expositions.   Generally  such  paltry 
sums  as  above  are  not  newsworthy.   A  $l6o,000,000  overrun 
would  be  more  like  it.   Actually  the  article  refers  to  a 
claim  filed  by  descendants  of  the  shipbuilder  over  a  disputed 
unpaid  bill  resulting  from  construction  of  a  Union  Navy  gun- 
boat during  the  Civil  War.   As  in  todays  similar  reports, 
the  ultimate  "success"  of  the  gunboat  -  or  failure  -  is  left 
unaddressed. 

In  the  conclusion  of  an  exhaustive  study  by  Dallen, 
Merkl  and  Zlatoper  of  the  Army's  ill-fated  main  battle  tank 
(MBT-70)  program,  it  is  noted  that  various  players  in  the 


weapons  acquisition  process  view  "success"  quite  differ*- 

1 
ently.    The  authors  observed  that  the  strength  of  "sponsor- 
ship" of   a  given  weapons  acquisition  may  Very  well  determine 
whether  or  not  the  system  eventually  becomes  part  of  the 
nation's  arsenal,  regardless  of  the  actual  need  or  degree 
of  problems  encountered  in  the  development  and  acquisition. 
The  study  authors,  therefore,  supported  the  "adoption  of  a 

set  of  'rational'  criteria  by  which  to  Judge  system  sue- 
2 

cess.  " 

THESIS  OBJECTIVES 

Improving  the  results  of  defense  systems  acquisition 
depends  on  learning  from  past  mistakes  and  past  successes. 
This  requires  consistent  evaluation  of  project  results  with 
as  much  objectivity  as  possible. 

The  purpose  of  this  thesis  is  to  investigate  the  possi- 
bility of  such  a  system  of  evaluation  of  the  degree  of  suc- 
cess in  naval  ship  acquisition.   The  objectives  then  are  to 
determine  for  naval  ship  acquisitions  a  criterion  for  succes 
and  investigate  the  feasibility  and  alternatives  for  such  a 
system. 


Footnotes  and  references  are  listed  at  the  ends  of  the 
chapters. 
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THE  COMPLEXITY  OF  TOE  HARDWARE 

Articles  on  naval  ship  acquisition  usually  begin  with 
acknowledgment  that  major  defense  weapons  systems  are  the 
most  technically  complex  of  any  in  existence.   Perhaps  the 
most  complex  of  weapons  systems  are  naval  ships. 

Ships  are  the  largest  mobile  objects  on  Earth  and  naval 
vessels  represent  an  integration  of  a  myriad  of  major  and 
minor  interlocking  systems  of  which  many  are  extremely  com- 
plex in  their  own  right.   A  nuclear  powered  aircraft  carrier 
is  over  1100  feet  long,  displaces  over  90,000  tons,  and  is 
propelled  at  speeds  in  excess  of  30  knots  by  over  200,000 
shaft  horsepower.   It  is  an  airport  with  a  capacity  of  100 
or  more  Jet  aircraft.   Additionally,  it  is  a  self  contained 
city  with  a  population  of  5,000  to  6,000  people.   Its  multi- 
reactor  nuclear  plant,  various  electronic  systems  and  air- 
craft launching  and  recovery  systems  take  years  to  design 
and  prove  before  they  are  ready  for  instiillation. 

THE  COMPLEXITY  OF  THE  PROBLEM 

The  complexity  of  the  hardware  in  question  is  perhaps 
only  matched  by  the  complexity  of  the  bureaucracy  and  pro- 
cess concerned  with  the  acquisition  of  the  hardware.   In 
World  War  II,  the  entire  Navy  Department  in  Washington, 
D.C,  charged  with  directing  a  war  effort  involving  at  th§ 
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peak  of  the  war  about  50,000  ships,  consisted  of  about  200 
people.   Today  there  are  in  excess  of  20,000  in  the  Washing- 
ton, D.C.,  metropolitan  area  employed  by  the  Navy.   The  ac- 
tive fleet  today  consists  of  less  than  500  ships. 

It  commonly  takes  ten  to  twelve  years  to  conceive, 
develop,  design  and  construct  a  new  U.S.  Navy  ship  class. 
The  problems  of  long  range  fiscal  and  engineering  forecast- 
ing and  planning  in  an  unstable  political  and  economic  envir- 
onment;  and  in  the  face  of  rapid  technological  advances,  can 
be  "mind  boggling"  as  one  Ship  Acquisition  Project  Manager 
observed  to  the  author. 

THESIS  OVERVIEW 

The  first  chapter  of  the  thesis  is  concerned  with  the 
three  rather  distinct  strategies  employed  in  defense  systems 
acquisition  since  World  War  II.   Though  certainly  not  a  com- 
plete treatment,  the  reader  should  gain  from  Chapter  I  con- 
siderable background  information  about  the  naval  ship  (and 
other  weapon  systems)  acquisition  process  and  structure. 
The  major  emphasis  is  on  several  new  systems  and  policies 
which  have  been  formulated  during  the  past  five  or  six  years, 
The  reader  interested  in  a  more  complete  analysis  of  the 
defense  weapons  acquisition  process,  particularly  during  the 

1960's,  is  referred  to  the  recent  and  excellent  book 

3 
Arming  America  by  J.  Ronald  Pox. 
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Chapter  II  is  concerned  with  the  case  for  a  comprehen- 
sive, consistent  evaluation  system  for  naval  ships.   The 
problem  of  acquisition  projects  outliving  acquisition  poli- 
cies and  strategies  is  discussed.   Then,  the  existing  over- 
emphasis on  "short-term"  goal  achievement,  and  the  reasons 
for  that  over-emphasis,  at  the  expense  of  longer  term,  stra- 
tegic goals,  is  investigated. 

It  is  concluded  in  Chapter  II  that  a  comprehensive  eval- 
uation system  could  help  shift  the  balance  of  goal  emphasis 
as  well  as  fill  the  need  for  a  centralized,  complete  histori- 
cal record  and  data  bemk  for  ship  acquisitions.   A  position  is 
taken  against  use  of  the  system  for  personnel  performance 
evaluation. 

Chapter  III  begins  with  noting  that  relevancy,  objectiv- 
ity and  feasibility  are  desirable  traits  of  a  success  evalua- 
tion system,  but  that  total  objectivity  (requiring  relevant, 
strictly  quantitative  criteria)  is  unobtainable  in  a  feasible 
evaluation  system  for  such  a  complex  system. 

The  specification  of  goals  and  establishment  of  their 
relative  priorities  from  the  need  requirements  are  noted  as 
being  prerequisites  to  the  evaluation  system.   A  definition 
of  success  is  also  required  and  a  definition  preference  for 
naval  ship  acquisitions  is  offered. 

The   remainder  of  Chapter  III  is  devoted  to  developing 
sets  of  criteria  for  the  three  major  goal  areas:  performance, 
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cost  and  schedule.   Finally,  two  different  approaches  to 
combining  the  individual  criteria  so  that  an  overall  assess- 
ment of  success  is  possible  are  discussed. 

A  manual  evaluation  model  is  outlined  and  the  reasons 
for  adopting  such  a  "low  resource  requirement"  approach  are 
offered.   The  scheme  is  similar  to  those  utilized  by  acquisi- 
tion projects  for  evaluating  contractor  bid  proposals. 

A  framework  for  the  development  of  a  much  more  sophisti- 
cated, computerized  simulation  model  is  presented  as  an 
alternative  to  the  manual  model.   This  approach  requires 
more  resources  to  develop  and  use,  but  offers  the  possibility 
of  a  more  accurate  and  objective  model  as  more  of  the  complex 
interdependencies  of  an  acqulsitibm can  be  accounted  for 
and  probabilistic  criteria  can  be  fully  utilized. 

T^e  final  conclusions  and  recommendations  include 

a)  the  success  evaluation  system  outlined  in  this 
thesis  is  imperfect  but  preferable  to  the  current  situation 
(i.e.  no  long  term  structured  system  at  all), 

b)  more  work  needs  to  be  done  in  constructing  evalna- 
tion  criteria  and  in  implementing  existing  and  planned  systems 
upon  which  the  proposed  evaluation  system  is  built, 

and, 

c)  the  further  specification  and  trial  implementation 
of  the  proposed  system  of  success  evaluation  is  recommended. 
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CHAPTER  I 
AN  OVERVIEW  OF  POST  WWII  U.S.  NAVAL  SHIP  ACQUISITION 

INTRODUCTION 

A  brief  overview  of  the  structure  and  process  of  naval 
ship  acquisition  in  the  United  States  since  World  War  II  is 
presented  in  this  chapter.   Three  rather  distinct  periods  are 
indortifiable  -  the  "Conventional"  Period  (until  the  early 
1960* s),  the  "Concept  Formulation/Contract  Definition"  or 
"Total  Package  Procurement"  Period  (also  called  the  "McNamara" 
Period  after  the  then  Secretary  of  Defense  Robert  S.  McNamara) 
which  began  in  the  early  1960's  and  ended  about  I969*  and  the 
"Current"  Period.   The  major  policies  and  characteristics  of 
the  three  periods  differ  considerably,  but  it  is  not  always 
possible  to  categorize  a  particular  ship  acquisition  project 
as  being  a  result  of  the  policies  of  any  one  particular  peri- 
od.  For  example,  the  nuclear  powered  aircraft  carrier  pro- 
ject (the  NIMITZ  class  of  ships)  was  conceived  during  the 
Conventional  Period,  continued  through  the  Concept  Formula- 
tion/Contract Definition  Period,  and  remains  an  ongoing  pro- 
ject.  The  project  has  characteristics  of  all  three  policy 
periods.   The  ships  were  designed  primarily  by  the  Navy  ("in 
house")  with  the  aid  of  a  design  agent,  typical  of  the  Con- 
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ventional  and  Current  Periods.   The  three  (or  four)  ships  of 
the  class  are  being  constructed  under  a  multi-ship,  multi- 
year  contract,  a  significant  aspect  of  the  Concept  Formula- 
tion/Contract Definition  Period  which  has  been  continued. 

Thus  as  the  different  periods  are  described,  it  should 
be  remembered  that  ship  projects  are  long  (often  ten  to 
twelve  years)  and  often  transcend  major  acquisition  policy 
shifts.   Additionally,  like  any  large  bureaucracy,  new  poli- 
cies and  strategies  from  top  management  (the  Secretary  of 
Defense)  often  do  not  take  effect  at  the  working  level  (the 
projects)  for  two  or  three  years,  if  at  mil. 

THE  BASIC  PROCESS 

Although  policies  and  the  organizational  structure 
for  designing  and  acquiring  ships  for  the  U.S.  Navy  have 
changed  over  the  years,  the  basic  process  remains  much  the 
same.   Also,  though  differing  in  details  and  nomenclature, 
the  acquisition  of  ships  is  similar  to  the  acquisition  of 
other  major  defense  systems.   A  need  is  identified;  a  re- 
quirement based  on  that  need  is  established;  a  weapon  sys- 
tem is  selected,  developed,  designed  and  constructed  to  fill 
the  requirement.   Sometimes  technological  breakthroughs 
motivate  a  new  acquisition  (termed  "technology  push")  but 
attempts  are  made  to  ensure  that  a  legitimate  need  exists 
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prior  to  development  and  construction  of  the  system. 

The  Navy  and  the  other  services  are  charged  with  "iden- 
tifying needs  and  defining,  developing  and  producing  systems 

1 
to  satisfy  those  needs."   Establishing  overall  acquisition 

policy,  passing  on  the  validity  of  needs,  and  monitoring  the 

performance  of  the  services  in  carrying  out  the  policy  is 

2 
the  purview  of  the  Office  of  the  Secretary  of  Defense. 

National  defense  policies  and  objectives  are  provided  by 
the  Secretary  of  Defense  and  translated  by  the  Joint  Chiefs 
of  Staff  into  military  policies  and  objectives.   Planning  and 
progrsimming  by  the  services  is  keyed  to  these  strategic  ob- 
jectives.  Evaluation  of  the  Joint  Chiefs  of  Staff  guidance 
can  lead  to  research  and  development  objectives  formulation 
by  the  services  to  satisfy  deficiencies  in  their  capabili- 
ties to  perform  their  respective  assigned  missions  and  . 

3 
roles.    The  Program  Objectives  Memorandum  is  the  budget  for 

this  effort.  Later,  Program  Objectives  Memoranda  will  con- 
tain the  weapons  systems  which  emerge  from  the  research  and 
development  efforts. 

The  Program  Objectives  Memoranda  are  part  of  the  Depart- 
ment of  Defense  Planning-Programming-Budgeting  System  (PPBS). 
Funding  for  weapons  systems  is  obtained  through  the  PPBS. 
However,  a  series  of  approvals  by  intra-service  councils 


T 

Notes  are  listed  at  the  end  of  each  chapter. 
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and  the  top  level  Defense  Systems  Acquisition  Review  Council 
is  currently  also  required  before  a  new  weapons  system  is 
constructed.   The  role  of  the  Defense  System  Acquisition 
Review  Council  will  be  discussed  later  in  the  chapter. 

THE  CONVENTIONAL  PERIOD 

At  the  end  of  World  War  II  the  U.S.  Navy  had  50,000 
ships  and  there  were  50  private  and  public  shipyards  work- 
ing at  or  near  capacity.   Much  of  the  fleet  was  "mothballed" 
(retired  from  active  service)  at  the  end  of  the  war  and  pro- 
duction virtually  ceased  at  the  shipyards.   During  the  Korean 
conflict  most  of  the  required  ships  were  reactivated  from 
the  "mothball  fleet".   It  was  not  until  1952  that  the  first 

significant  new  shipbuilding  since  World  War  II  -  31  major 

5 
ships  -  was  directed. 

The  major  policies  and  practices  were  characterized 
by  an  iterative  design  process  accomplished  by  the  Navy 
"in  house"  or  by  an  independent  design  agent  up  to  and  includ- 
ing a  complete  construction  bid  package,  little  documenta- 
tion, major  emphasis  on  ship  performance,  the  splitting  of 
the  production  contracts  between  two  or  more  shipbuilders, 
and  little  involvement  by  the  Office  of  the  Secretary  of 
Defense.   The  acquisition  process  was  basically  decentral- 
ized to  the  service  level. 
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Initially,  the  entire  design  and  procurement  effort 
would  be  coordinated  by  a  few  individuals.   They  relied 
almost  completely  on  the  fixnctional  organizations  to  perform 
the  necessary  design  and  acquisition  work  required.   Differ- 
ent self-contained  organizations  would  be  responsible  for 
various  systems  on  the  ship.   The  "Gun  Club"  (Bureau  of 
Ordnance)  was  responsible  for  weapons  for  example.   Later, 
starting  with  the  Polaris  and  other  ballistic  missile  pro- 
grams, the  trend  was  towards  project  manager  type  organiza- 
tions with  varying  degrees  of  self  containment. 

The  reasons  for  spreading  the  production  contracts 
between  several  shipbuilders  are  not  known  for  certain  but 
probably  included  the  desires  for  rapid  delivery  of  the  ship 
class  and  preservation  of  the  production  base  of  the  ship- 
yards.  Alternatively,  the  primary  reason  may  have  simply 

been  regional  political/economic  pressures. 

6 
As  depicted  in  Figure  1-1  ,  the  "Conventional"  approach 

typically  consisted  of  "in-house"  talent  formulating  a  ship 

concept.   Iliis  activity  included  cost  and  feasibility  studies 

and  advance  research  and  development  as  necessary.   Assuming 

budgetary  approval,  increasingly  refined  and  detailed  design 

stages,  termed  "Preliminary  Design"  and  'Contract  Design" 

followed.   The  approach  was  basically  evolutionary  and  util- 

7 
ized  minimal  systems  analysis  techniques. 
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The  resulting  product  was  a  complete  bid  package,  in- 
cluding complete  contract  plans  and  specification.   The 
basic  bid  package  could  result  in  any  number  of  procurement 
contracts.   Lead  ships  (the  first  of  a  class)  were  often 
constructed  in  Navy  shipyards.   The  amphibious  ships  LPD 
7  through  15  were  built  under  four  separate  contracts  by 
two  shipbuilders.   Exclusive  of  the  costs  of  changes  to  the 
contract,  these  ships  were  delivered  to  the  Navy  at  an 
average  125?^  of  the  initial  contract  price  (fixed  price)  and 
an  average  27  months  behind  schedule.    Escalation  due  to 
inflation  and  claims  against  the  government  by  the  contrac- 
tors accounted  for  most  of  the  average  2^%   cost  overruns. 

The  above  results  became  increasingly  typical.   Low  or 
negative  profit  performance  by  the  contractors  is  thought  to 
have  precipitated  many  of  the  claims.   The  basis  of  conten- 
tion in  the  claims  was  usually  a  dispute  over  interpretation 
of  the  often  complex  and  detailed  contract  specifications. 
Additionally,  production  facilities  were  becoming  antiquated 
and  uncompetitive  in  the  world  market.   Support  of  the  ships 
was  costly  and  often  inadequate  due  to  a  lack  of  standardi- 
zation between  ships  of  a  given  class  and  between  classes 
of  ships. 
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THE  CONCEPT  FORMULATION/CONTRACT  DEFINITION 
TOTAL  PACKAGE  PROCUREMENT  PERIOD 

A  radically  different  approach  to  weapons  design  and 
acquisition  was  formulated  in  the  early  1960's  by  the 
Office  of  the  Secretary  of  Defense  under  Robert  S»  McNamara. 

The  new  approach  centralized  major  decision  making  authority 

9 
to  the  Office  of  the  Secretary  of  Defense.   Objectives  were 

a)  optimization  of  cost  effectiveness  by  using  systems 
analysis  techniques, 

b)  reduction  or  elimination  of  contractor  claims 
against  the  government  by  using  contractor  prepared  perform- 
ance oriented  specifications  vice  government  imposed  detailed 
specifications , 

c)  reduction  of  cost  overruns  by  transferring  finan- 
cial risk  to  the  contractors  for  the  design  and  acquisition 
phases  through  the  use  of  fixed  price  contracts^; j 

d)  significant  capitalization  increases  in  shipbuild- 
ing facilities  by  using  multi-ship,  multi-year  contract 
awards  to  a  single  shipbuilder  that  were  expected  to  pro- 
vide long  term  financial  security  enabling  large  scale  capi- 
talization and  force  expansion  of  facilities  due  to  delivery 
schedule  demands, 

e)  reduce  xmique  systems  and  subsystems  proliferation 
resulting  from  split  production  contracts. 
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f )  introduce  producability  and  innovation  into  the 
designs  by  having  the  production  contractor  design  the 
system, 

g)  lower  acquisition  costs  by  taking  advantage  of  the 
"Learning  Curve"  effect  through  single  producer  serial  pro- 
duction {The   "Learning  Curve"  had  long  been  thought  not  to 
apply  to  shipbuilding  but  several  studies  conducted  during 
the  1950 's  and  1960's  indicated  otherwise.), 

and, 

h)  arrive  at  more  accurate  total  cost  estimates  and 
reduce  poor  ship  support  by  making  the  contractor  responsi- 
ble for  all  on  board  systems,  crew  training,  initial  repair 
parts,  and  support  facilities.   (This  policy  is  known  as 
"Total  Package  Procurement".   Strictly  speaking  neither  of 
the  Navy  acquisitions  being  acquired  under  Concept  Formula- 
tion/Contract Definition  policies  are  "Total  Package  Pro- 
curement" as  there  is  government  furnished  equipment  involv- 
ed.) 

A  project  manager  type  organization  was  directed  for 
10 
all  major  programs   .   As  outlined  in  Figure  I-l,  the  ser- 
vices would  still  conduct  advanced  research  and  development 
and  identify  the  desired  performance  characteristics  of  the 
new  weapon  system  during  the  Concept  Formulation  stage. 
Assuming  approval  by  the  Secretary  of  Defense,  a  Contract 
Definition  period  would  follow.   A  "Request  for  Proposal" 
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was  prepared  by  the  service  and  issued  to  selected  shipbuil- 
ders (when  more  than  one  were  thought  capable)  to  prepare 
design  analyses  based  on  the  specified  performance  charac- 
teristics.  The  Request  for  Proposal  contained  both  manda- 
tory and  desirable  performance  specifications  and  were  sup- 
posed to  "encourage  alternatives  and  stimulate  initiative 

11 
and  creativity  by  the  contractors". 

After  evaluation  of  the  proposals  by  the  service,  nor- 
mally two  or  more  contractors  were  given  fixed  price  con- 
tracts to  develop  a  complete  shipbuilding  proposal.   Required 
in  these  proposals  were  contract  plans  and  specifications, 

detailed  construction  plans,  management  plans  and  a  complete 

12 
analysis  of  the  Life  Cycle  Costs.    Life  Cycle  Costs  meant 

the  total  costs  of  acquisition  and  ownership  -  development, 

13 
production,  deployment,  operation  and  maintenance. 

Total  system  tradeoffs  were  encouraged  "to  obtain, 
within  the  mission  and  performance  envelopes,  an  optimum 
balance  between  total  cost,  schedule,  and  operational  effect- 
ivene*  for  the  system".   Operational  effectiveness  was  defin- 
ed to  include  "pure"  performance,  reliability,  maintainabil- 
ity and  all  other  factors  which  affected  the  effectiveness  of 
the  new  system.   The  system  was  defined  to  include  the  re- 
quired facilities,  personnel,  data,  training  equipment  as 

14 
well  as  the  the  hardware  itself. 
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No  longer  than  six  months  was  allowed  for  the  Contract 
Definition  phase.   This  was  followed  by  a  Source  Selection 
process  during  which  a  detailed  analysis  of  the  proposals 
was  conducted  by  the  procuring  service.   Negotiation  was  con- 
ducted with  one  or  more  of  the  contractors  and  transfusion 
of  parts  of  one  proposal  into  another  was  permitted.   At  the 
end  of  the  evaluation  period  a  recommendation  was  sent  to 
the  Office  of  the  Secretary  of  Defense  to  award  a  multi- 
year,  multi-ship  procurement  contract  to  the  contractor  with 
the  best  proposal,  to  conduct  further  Contract  Definition, 
or  to  defer  or  abandon  the  effort  (unlikely).   The  single 
contract  awarded  was  fixed  price,  with  or  without  incentive 
clauses. 

The  Navy  conducted  three  Concept  Formulation/ Contract 
Definition  ship  competitions.   The  Past  Deployment  Logistics 
Ship  (FDL  class)  was  not  funded  by  Congress.   The  Amphibious 
Helicopter  Assault  ships  (LHA  class)  and  the  SPRUANCE  class 
destroyers  (DD-963  class)  are  currently  under  construction. 
Litton  Industries  won  all  three  competitions. 

CONCEPT  FORMULATION/CONTRACT  DEFINITION  RESULTS 

"Frankly  gentlemen,  in  defense  procurement,  we 
have  a  real  mess  on  our  hcuids. 

The  Honorable  David  Packard 
Deputy  Secretary  of  Defense, I970 

It  is  too  early  to  evaluate  the  final  results  of  the 

DIA  and  DD-963  shj.p  acquisitions.  The  USS  SPRUANCE  (DD-963) 

24 


is  the  first  ship  delivered  under  either  contract  and  was 
accepted  by  the  Navy  in  1975.   However,  the  acquisitions  have 
been  beset  by  much  the  same  problems  that  have  characterized 
defense  weapons  systems  procurement  the  last  two  decades  - 
acquisition  cost  and  schedule  overruns  of  large  proportions. 
This  is  particularly  true  of  the  first  of  the  contracts, 
the  LHA  class.   The  first  LHA  is  yet  to  be  delivered  auod  is 
several  years  behind  the  original  schedule.   A  large  claim 
against  the  government  by  Litton  over  the  contract  is  pend- 
ing and  the  contract  has  already  been  renegotiated  (upwards 
in  price  and  longer  delivery  times). 

By  the  late  1960's,  cost  and  schedule  overruns  and 
performance  shortfalls  of  new  major  weapons  systems  were 
daily  newspaper  fare.   In  1971  the  Assistant  Secretary  of 

Defense  for  Financial  Management  (Comptroller)  conducted  a 

16 
survey  of  35  major  development  and  production  programs. 

Only  two  of  the  programs  were  found  to  be  on,  or  ahead  of, 
schedule.   That  sajne  year  the  General  Accounting  Office 
made  a  survey  of  61  weapon  systems  and  found  that  cost  esti- 
mates for  them  had  increased  $33.4  billion  over  the  initial 

17 
estimates.    Contractor  costs  soared  and  profits  plummeted. 

Huge  cost  and  schedule  problems  in  the  C-5A  aircraft  were 

overshadowed  in  1971  when  cracks  developed  in  the  engine 

18 
mountings.    The  term  "contractor  bailout"  became  a  house- 
hold word  as  one  producer  after  another  threatened  to  cease 
production  unless  relief  from  the  fixed  price  contracts  was 
provided.  25 


By   1970  a  number  of  studies  had  found  serious  flaws 
in  the  management  of  the  weapons  acquisition  process. 
The  President's  Blue  Ribbon  Panel,  The  National  Security 
Industrial  Association,  the  Defense  Science  Board  Task 
Force  on  Research  and  Development  Management,  the  Aero- 
space Industries  Association,  and  the  General  Accounting 
Office  conducted  studies  which  all  sounded  very  much  the 

same  theme.   A  March  1970  Logistics  Management  Institute 

19 
report  summarized  the  most  common  criticisms  as  follows: 

"First  is  the  observation  that  the  weapons 
acquisition  process  apparently  is  out  of 
control.   Initial  time  and  cost  estimates  - 
and  even  updated  estimates  -  cannot  be  depen- 
ded upon.   Mandatory  engineering  changes  arise 
continually  throughout  the  process.   Manage- 
ment information  and  control  systems  do  not 
identify  impending  problems  in  time  for  pre- 
ventive action  to  be  taken. 

Second  is  the  claim  that  bargaining  positions 
are  unbalanced;  first  one  side,  the  the  other 
has  the  advantage.   The  theory  of  counter- 
vailing pressures  acting  to  produce  fair  and 
realistic  contract  terms  does  not  hold.   With 
emphasis  on  economies  of  scale  and  series  pro- 
duction there  are  only  a  small  number  of 
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weapon  systems  competitions  each  year  and 
prospective  contractors  believe  that  their 
very  existence  may  be  Jeopardized  by  failure 
to  win.   Hence  the  Department  of  Defense  (DOD) 
is  in  the  dominant  position  and  can  compel 
an  unreasonable  bargain.   Following  award 
of  the  contract,  the  DOD,  coimnitted  to  the 
timely  success  of  the  program,  is  in  the 
weaker  position  as  the  sole  source  contrac- 
tor negotiates  for  contract  changes,  prod- 
uct acceptance,  and  follow-on  business. 
Third  is  that  incentives  both  for  eff in- 
dent operation  and  for  candor  about  expec- 
tations are  lacking.   Heavy  reliance  on 
historical  costs  in  pricing,  lack  of  ade- 
quate consideration  of  capital  required 
in  negotiating  profit  rates,  and  the  high 
risk  of  low  future  utilization  of  contrac- 
tor owned  facilities  impede  investment  and 
modernization  of  plant.   The  hazard  to  pro- 
gram survival,  of  high  cost,  long  duration, 
or  looming  technical  difficulties,  as  each 
program  competes  with  others  in  and  out  of 
the  DOD,  motivates  extreme:  optimism  by 
DOD  and  contractor  personnel  alike. 
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Fourth  are  allegations  of  confusion,  con- 
nivance, and  deception  by  the  DOD-con- 
tractor  combination.   Close  cooperation 
and  common  interest  are  held  in  contrast 
to  the  arm's  length  relationship  prefer- 
red by  much  of  regulation  and  policy. 
Policy  notwithstanding,  the  military 
departments  receive  advice  and  assistance 
from  prospective  contractors  in  prepara- 
tion of  Requests  for  Proposals.   Contrac- 
tors receive  aid  from  government  person- 
nel in  performance  of  contracts.   Contracts 
fail  as  instruments  of  control." 
On  May  28,  1970,  Deputy  Secretary  of  Defense  David 
Packard  issued  a  memorandum  which  stated  that  the  Total 
Package  Procurement  approach  to  developing  and  acquiring 

major  weapon  systems  was  unsatisfactory  and  that  a  new 

20 
policy  would  soon  be  established.    The  Navy  was  still 

years  away  from  delivery  of  its  first  ship  procured  under 
the  cancelled  policy.   Whether  or  not  the  long  term  objec- 
tives of  the  Concept  Formulation/ Contract  Definition/Total 
Package  Procurement  Period  will  be  achieved  for  the  Navy 
ships  remains  to  be  seen.   It  is  a  fact  that  the  sole  source 
multi-year  contracts  resulted  in  the  construction  by  Litton 
of  a  new  shipyard  at  Pascagoula,  Mississippi.   Interviews 
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conducted  by  the  author  with  project  personnel  brought  out 
the  beliefs  of  several  that  the  policies  were  never  given  a 
fair  chance.   This  resulted  from  the  failures  of  the  F-111 
and  C-5A  aircraft  procurements  in  particular  and  the  problems 
that  Litton  was  experiencing  in  developing  an  adequate  design 
and  production  force  and  in  making  the  new  facility  operas 
tionally  efficient. 

THE  CURRENT  PERIOD 

The  major  policies  and  trends  of  new  ship  acquisition 
since  the  demise  of  Concept  Rjrmulat ion/ Contract  Definition 
include 

a)  emphasis  on  constrained  design  ("Design- to- Cost"), 

b)  emphasis  on  proven  hardware  ("Ply-before-you-Buy"), 

c)  required  review  and  approval  to  proceed  by  the 
Defense  Systems  Acquisition  Review  Council  at  key  milestones, 

d)  a  prohibition  against  Total  Package  Procurement 
(linkage  of  system  development  and  production  under  a  single 
contract), 

e)  improvement  of  the  quality  and  validity  of  cost 
estimates, 

f )  flexibility  in  contract  type  and  liberalization 
of  contract  escalation  (due  to  inflation)  clauses, 

g)  use  of  contractor  aided  "in-house"  ship  design. 
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and, 

h)  tailoring  of  acquisition  apprtjAches  to  each  pro- 
ject. 

"DESIGN- TO-COST" 

21 
The  cancellation  of  the  key  top  level  policy  directive 

for  the  Concept  Formulation/ Contract  Definition  period  in 
1970  left  a  guidance  void  that  was  not  formally  filled  until 
the  issuance  of  Department  of  Defense  Directive  5000>.l, 
"Acquisition  of  Major  Defense  Systems",  on  July  13,  1971. 
It  was  during  this  same  period  that  then  Chief  of  Naval 
Operations  Zumwalt  directed  the  rapid  development  of  a  large 
class  of  austere,  relatively  Inexpensive  Guided  Missile  Fri- 
gates (FFG  class,  formerly  the  Patrol  Frigate  or  PF  class) 

22 
to  bolster  the  size  of  the  rapidly  diminishing  fleet. 

After  a  year  of  feasibility  studies >  Admiral  Zumwalt 

directed  that  the  design  would  not  violate  constraints  which 

were  set  on  the  average  "follow"  (after  the  first)  ship 

23 
acquisition  cost  and  fully  loaded  ship  displacement.    He 

later  added  a  third  constraint  on  the  maximum  number  of 

24 
accomodations  on  the  ship.    Performance  capability  above 

the  minimum  specified  was  to  be  traded  off  to  stay  within 
the  constraints.   This  method  of  ship  design,  commonly  termed 
"Design- to-Cost",  was  revolutionary  to  the  Navy,  but  is  com- 
mon in  industry  for  new  product  development. 
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A  major  program  consideration  of  current  acquisition 
policy  is  that  "discrete  cost  elements  (e.g.  unit  produc- 
tion cost,  operating  and  support  cost)  shall  be  translated 

25 
into  "design  to"  requirements".    In  October  1973  the  major 

services'  material  commands  Jointly  issued  a  "Joint  Design- 

26 
to-Cost  Guide".    This  directive  requires  that  "Design-to- 

Cost  be  used  for  major  systems  except  that 

a)  "Major  weapons  systems  which  for  reasons  of 
national  security  necessarily  push  the  state-of-the-art 
should  stress  performance.   In  such  cases,  unit  production 
cost  goals  will  be  applied  and  tracked  but  will  be  subordi- 
nate to  performance  goals." 

and, 

b)  "Major  weapons  systems  where  the  assessed  need 

is  sufficiently  urgent  should  strtps  the  schedule.   In  such 

cases,  unit  protluction  cost  goals  will  be  applied  and  tracked 

but  will  be  subordinate  to  schedule  goals." 

Implementation  of  the  above  exceptions  requires  Secretary  of 

Defense  approval.   The  TRIDENT  strategic  ballistic  missile 

submarine  class  is  such  an  example. 

Historically,  performance  requirements  for  new  ships 

had  been  dictated  by  the  Chief  of  Naval  Operations  to  the 

28 
material  command  in  brief  "single  sheet  characteristics". 

These  were  used  by  the  material  command  "to  develop  prelim- 
inary designs  and  cost  estimates  leading  to  more  detailed 
characteristics  statements,  and  ultimately  to  procurement 

31 


29 
specifications."    Costs  were  considered  but  were  usually 

secondary  to  maximizing  performance.   As  "Design- to-Cost" 
elevated  the  Importance  of  acquisition  cost  to  the  same  level 
as  performance  in  the  design  process,  a  new  iterative  perfor- 
mance-cost tradeoff  dialogue  between  the  customer  (Chief  of 
Naval  Operations)  and  producer  (Chief  of  Naval  Material) 
organizations  was  required.   "Top  Level  Requirements  and 
Top  Level  Specifications  for  the  development  of  naval  ships". 
Chief  of  Naval  Operations  Instruction  9010.300,  was  issued 
early  in  1974.   It  details  a  procedure  which  provides  for  a 
working  group  to  develop  the  performance  parameters  for  a 
baseline  ship  which  will  meet  the  established  mission  re- 
quirements.  The  group  also  specifies  allowable  variations 

in  performance  parameters  and  alternative  system  selections 

30 
for  the  ship  class. 

After  a  period  of  Feasibility  Studies  during  which  the 
impact  of  the  alternative  performance  parameters  and  systems 
selections  are  evaluated,  the  Chief  of  Naval  Operations 
organization  starts  preparation  of  a  Draft  Top  Level  Require- 
ments document.   This  document  is  revised  as  the  Naval  Mater- 
ial Commajid  (parent  command  of  the  Naval  Sea  Systems  Command 
for  whom  acquisition  projects  directly  work)  develops  a  con- 
ceptual design  for  the  ship  class  and  provides  cost  and 
design  information  to  the  Chief  of  Naval  Operations. 
Assuming  approval  by  the  Chief  of  Naval  Operations  to  proceed 
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with  a  selected  design,  a  "Conceptual  Baseline"  and  a  "Cost 
Goal"  for  the  average  follow  ship  acquisition  costs  are  pre- 
sented to  the  Defense  Systems  Acquisition  Review  Council 
which  is  composed  of  high  level  officials  in  the  Office  of 
the  Secretary  of  Defense.   If  council  and  Secretary  of 
Defense  approval  is  given  to  proceed  into  "Preliminary 
Design",  a  Draft  Top  Level  Specifications  document  is  started 

by  the  Naval  Material  Command  (actually  the  acquisition 
project).   This  companion  document  to  the  Top  Level  Require- 
ments "translates  the  Top  Level  Requirements  into  a  physical 

31 
ship  description".    The  contents  of  the  two  documents  are 

discussed  in  Chapter  III.   The  process  phasing  is  presented 

32 
graphically  in  Figure  1-2. 

The  Naval  Sea  Systems  Command  recently  issued  a  "Design- 

33 
to-Cost  Guide  for  Ship  Acquisition"   .   Ihis  directive  spe- 
cifies that  the  procedure  of  "Desgin- to-Cost"  will  be  utili- 
zed for  all  ship  acquisitions  for  which  a  Top  Level  Require- 
ments document  is  prepared.   It  exempts,  as  do  most  ship 
acquisition  directive^  nuclear  propulsion  plants  and  nuclear 
support  facilities  under  the  cognizance  of  Admiral  Hyman  G. 
Rickover,  Director  of  Naval  Reactors.   The  Chief  of  Naval 
Operations  has  stated  that  a  Top  Level  Requirements  and  Top 

Level  Specifications  will  be  developed  for  "all  new  ship  de- 

34 
signs  except  fleet  ballistic  missile  submarines." 
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"FLY-BEFORE- YOU-BUY" 

It  had  been  intended  that  before  permission  to  proceed 
into  Contract  Definition  would  be  granted,  "primarily  engine- 
ering rather  than  experimental  effort"  would  be  required  and 
the  technology  needed  would  be  "well  in  hand".    However 
the  large  performance  shortfalls,  schedule  delays  and  cost 
increases  referred  to  in  the  previous  section  were  at  least 
partially  a  result  of  overly  optimistic  estimates  of  ultimate 

system  capabilities  and  the  time  required  to  design  and 
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perfect  them.    There  had  been  a  great  deal  of  reliance  on 

"paper  studies"  rather  than  on  actual  performance  demonstra- 
tions.  Thus,  a  major  program  consideration  of  the  Current 
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Period  is  that 

"Programs  shall  be  structured  and  resources 

allocated  to  ensure  that  the  demonstration 

of  actual  achievement  of  program  objectives 

is  the  pacing  function. " 

A  supporting  Department  of  Defense  directive  was  issued  in 

January  of  1973  to  establish  test  and  evaluation  policy  for 
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the  acquisition  of  defense  systems.    This  policy  emphasizes 

a)  the  need  to  conduct  test  and  evaluation  as  early 
as  possible  and  to  continue  it  as  long  as  necessary  to  elim- 
inate risks, 
and. 
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b)  acquisition  schedules  will  be  based  on  the  comple- 
tion of  test  and  evaluation  milestones  prior  to  key  decision 
points  (Defense  Systems  Acquisition  Review  Council  approval 
points.   See  Figure  1-2.) 

The  key  element  which  has  grown  from  recognition  of  the 
need  for  increased  test  and  evaluation  during  the  acquisition 
process  has  been  prototyping.   This  is  commonly  known  as  the 
"Ply-before-you-Buy"  policy  and  is  currently  being  used  in 
the  acquisition  of  new  aircraft  and  in  the  Navy's  surface 
effect  ships  and  NATO  patrol  hydrofoil  programs.   It  is  not 
feasible,  however,  to  build  and  evaluate  prototypes  prior  to 
beginning  follow  ship  design  and  production  for  large  ships. 
The  time  required,  small  numbers  involved  (one  or  two  in  some 
cases)  and  threat  of  obsolescence  dictate  a  modified  "KLy- 
before-you-Buy"  approach. 

The  Guided  Missile  Frigate  Program  developed  a  plan 
which  provided  for 

a)  early  construction  of  Land  Based  Test  Sites  for 
complete  propulsion  and  combat  systems 

and, 

b)  a  delay  of  two  years  between  construction  contract 
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awards  for  the  lead  and  follow  ships. 

This  plan  permits  testing  of  the  two  major  "high  risk" 

subsystems  prior  to  installation  on  the  lead  (first)  ship  and 

time  to  incorporate  changes  resulting  from  the  entire  test 
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and  evaluation  program  into  the  design  of  the  follow  ships. 
The  Land  Based  Test  Sites  will  also  be  used  for  crew  traiifp-  . 
ing. 

THE  DEFENSE  SYSTEMS  ACQUISITION  REVIEW  COUNCIL 

The   Defense  Systems  Acquisition  Review  Council  (DSARC) 

was  established  in  the  Office  of  the  Secretary  of  Defense 
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by  Deputy  Secretary  of  Defense  David  Packard  in  May,  I969. 

The  purpose  of  the  council  reviews  is  to  d^irwluttt* Ath^fcughly 

the  status  of  major  defense  systems  acquisition  projects  at 

three  critical  milestones  which  generally  occur  for  ship 

acquisitions  as  indicated  in  Figure  1-2. 

Formal  documentation  for  the  DSARC  reviews  and  decisions 

is  provided  by  the  project  prepared  Decision  Coordinating 

Paper  (formerly  Development  Concept  Paper).   It  is  a  summary 

document  of  not  more  than  twenty  pages  that  records  the 

primary  information  on  a  program  including  the  thresholds, 

issues  and  risks,  need,  the  alternatives,  the  reviews, 

rationale  for  the  decision,  and  affordability.   When  signed 

by  the  Secretary  of  Defense,  it  is  the  authority  for  the 

service  to  proceed  with  the  program  (or  whatever  action  the 

Secretary  of  Defense  directs).   His  decision  sets  the  limits 

of  authority  (thresholds)  within  which  the  project  is  obliga- 
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ted  to  stay. 
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A  long  series  of  Intra-service  briefings  and  reviews 
is  generally  required  of  an  acquisition  project  prior  to  a 
DSARC  presentation. 

COST  ESTIMATING 

By  1970,  the  credibility  of  the  Department  of  Defense 
concerning  system  acquisition  cost  estimates  was  extremely 
low.  When  the  new  acquisition  policy  directive  was  issued 
by  Mr.  Packard  in  1971,  it  directed  that  "Traceability  of 

estimates  and  costing  factors  be  maintained".   In 

December^  1971,  Mr.  Packard  issued  a  memorandum  requiring 
incorporation  of  independent  (from  program  proponents)  para- 
metric cost  analysis  in  Defense  Systems  Acquisition  Review 
Council  presentations. 

Parametric  cost  estimates  are  "based  on  quantified 
relationships  between  the  cost  and  physical  performance 
characteristics  of  past  systems  for  initial  program  decle- 
ions".    Engineering  cost  estimates  (normally  used  by  acqui- 
sition projects)  are  based  on  detailed  design  information. 
Cost  Information  Reports  (now  Contractor  Cost  Data  Reports) 
submission  requirements  have  been  written  into  production 
contracts  in  order  to  build  a  data  base  from  which  to  develop 
parametric  cost  estimating  relationships. 

Pox  discusses  a  third  estimating  alternative  for  use  in 

46 
preparing  contract  negotiation  objectives.    This  technique, 
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known  as  "should  cost"  analysis  had  its  beginnings  in  the 
civilian  economy  and  was  developed  in  the  early  1960'8  for 
the  Department  of  Defense,  under  the  direction  of  then  Defense 
Comptroller  Robert  N.  Anthony.   Pox  describes  the  process  as 
follows 

"This  technique  begins  with  an  in  depth  analysis 
of  a  contractor's  management^  cost  estimating^  and 
production  practices  to  identify  and  measure  the 
effects  of  poor  performance.   From  a  baseline  devel- 
oped by  eliminating  costs  resulting  from  inefficien- 
cies, projections  are  made  for  the  current  procure- 
ment.  The  objective  is  to  foster  improved  industrial 
practices  by  setting  realistic  contract  prices." 
Under  the  approach,  a  team  of  experts  in  engineering, 
pricing,  auditing,  procurement  and  management  spend  several 
weeks  at  the  contractor's  facility.   They  review  in  detail 
the  company's  operations  and  procedures.   Standard  industrial 
engineering  techniques  are  used  to  uncover  inefficiencies. 

Pox  cites  several  examples  of  should-cost  analyses 
successes  in  reducing  costs.   He  also  notes  that  it  is  dif- 
ficult to  achieve  high  quality,  factual  investigations  and 
that  there  had  been  considerable  resistance  from  within 
contractor  organizations  and  from  within  the  Department  of 
Defense. 

A  Cost  Analysis  Improvement  Group  (CAIG)  was  established 
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in  January  1972  in  the  Office  of  the  Secretary  of  Defense 

"to  review  the  estimates  presented  and  to  develop  uniform 

47 
criteria  to  be  used ".     5y  the  time  a  permanent  char- 
ter for  the  CAIG  was  issued  in  1973,  its  responsibilities 
had  grown  to  include,  interalia,  identification  of  where 

efforts  are  needed  to  improve  the  cost  estimating  capabili- 

48 
ties  of  the  services.     Improvement  of  the  validity  and 

tracking  of  cost  estimates  has  recently  been  addressed  by 

the  Secretary  of  the  Navy,  Chief  of  Naval  Operations,  and 
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Chief  of  Naval  Material  directives.     None  of  the  directives 

refer  to  the  "should-cost"  technique  of  cost  estimating. 

A  discussion  of  the  provisions  of  the  directives  is  contained 

in  Chapter  III.   Figure  1-3  shows  the  responsibilities  for 

ship  cost  estimating  and  the  cost  estimating  organizational 
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relationships  within  the  Navy. 


CONTRACTOR  COST  AND  SCHEDULE  CONTROL 

As  more  and  more  contractors  failed  to  perform  under 
the  Concept  Formulation/ Contract  Definition  Period  fixed 
price  contracts,  the  need  for  increased  government  engage- 
ment with  the  contractors  was  realized.   The  fixed  price 
contracts  and  lack  of  government  involvement  in  the  design 
supposedly  transferred  any  financial  risks  from  the  govern- 
ment to  the  contractor  and  thus  the  role  of  the  acquisition 
projects  was  basically  that  of  monitoring,  with  little  con- 
trol leverage.  2io 
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The  current  top  level  acquisition  policy  directive  spee 

cifles  that  "Contract  type  shall  be  consistent  with  all 

program  characteristics  including  risk,"  Also  stipulated  is 

that  "Cost  type  prime  and  subcontracts  are  preferred  where 

51 
substantial  development  effort  is  involved."   The  use  of 

cost  type  contracts  opens  up  the  possibility  of  increased 

government  engagement  with  the  contractor. 

Apart  from  the  lack  of  governmental  control  leverage 

resulting  from  fixed  price  contracts,  attempts  at  effective 

contractor  cost  and  schedule  control  by  the  projects  has 
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historically  been  hampered  by 

a)  a  reluctance  of  the  contractors  to  share  what  it 
considers  to  be  proprietary  information, 

b)the  preoccupation  of  project  managers  with  the 
annual  funding  approval  process  and  the  continuity  of  funds 
flow  (funds  control)  as  opposed  to  cost  control, 

c)  the  proliferation  of  various  information  and  cost 
control  systems'  imposed  on  contractors  by  the  different 
services  and  projects  (validity  of  the  information  was  often 
lost  in  the  translation  from  the  contractor's  system  to  the 
government  imposed  system(s)), 

d)  the  lack  of  inclusion  in  the  reporting  systems  of 
the  budgeted  value  of  work  performed, 

e)  improper  allocation  of  contractor  costs  between 
overhead (indirect)  and  direct  costs. 
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f)  inability  of  the  project  personnel  to  evaluate 
the  detailed  information  they  require  of  the  contractor,  and 
lack  of  correspondence  between  reported  data  and  the  contrac- 
tor's own  data, 

g)  retroactive  changing  of  finemcial  plans  to  conform 
to  work  performed  to  date  (the  "rubber  baseline") 

and, 

h)  contractor  use  of  nonintegrated  work  breakdown 
structures  and  nonintegrated  charts  of  cost  accounts  (sum 
of  budget  dollars  for  work  at  one  level  nay  exceed  budget 
at  next  higher  level). 

In  a  survey  conducted  during  the  1960's,  Pox  found  that 

most  program  managers  were  satisfied  if  their  funds  control 

reports  indicated  that  funds  were  being  expended  at  the  plan- 
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ned  monthly  rate  and  their  PERT  networks  reports   showed  no 

54 
significant  schedule  slippage.    Schedule  network  reporting 

based  on  starts  rather  than  completion,  untimely  or  inaccur^ 
ate  reporting,  and  the  lack  of  performance  of  scheduled  non- 
critical  path  work  all  served  to  build  in  cost  overruns  which 
often  went  undiscovered  until  it  was  too  late  to  take  any 
meaningful  cost  or  schedule  control  action. 

In  1967  the  Department  of  Defense  issued  a  directive 
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title  "Performance  Measurement  for  Selected  Acquisitions". 
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The  system  is  summarized  as  follows 

"Part  One  of  the  program  requires  that  contractors 
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use  internal  planning  and  control  systems  that 

meet  minimum  government  criteria.   These  criteria 

are  called  the  'Cost  and  Schedule  Control  Systems 

Criteria'  (CSCSC). 

Part  Two  of  the  program  requires  that  contractors 

regularly  submit  Cost  PerCormance  Reports  which 

contain  information  on  the  budgeted  value  of  work 

performed  to  date.   (The  criteria  themselves  do 

not  require  the  submission  of  any  reports  to  the 

government,  but  specify  the  reporting  capabilities 

which  contractors'  internal  systems  must  have,  and 

the  types  of  data  which  the  systems  should  be  able 

to  produce.)   The  contractor  is  free  to  design  his 

internal  planning  and  control  systems  to  correspond 

to  the  manner  in  which  he  organizes  his  work  units 

and  assigns  responsibility  for  performing  work." 

An  important  concept  in  the  reporting  criteria  is  the 

contract  cost  status  reports  must  be  based  strictly  on  the 

57 
number  of  Jobs  completed  to  date. 

Five  years  after  the  CSCSC  had  been  developed,  only 

16  defense  contractors  had  been  certified  as  complying  with 

the  criteria.   The  Navy  was  singled  out  by  the  Senate  Armed 

Service  Committee  as  being  particularly  slow  in  implementing 
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the  new  system.    In  I97I  an  additional  twenty  contractors 

complied  with  the  criteria  and  all  three  services  were  act- 
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ively  implementing  the  program  and  training  personnel  in  its 

59 
use. 


CURRENT  PERIOD  RESULTS 

It  is  certainly  too  early  to  make  amy  factual  observa- 
tions about  the  results  achieved  by  the  reforms  instituted 
since  1969-70  in  the  ship  acquisition  process.   The  Guided 
Missile  Frigate  program,  as  indicated,  pioneered  many  of 
these  reforms  in  the  Navy.   In  addition  to  those  aspects  of 
the  Guided  Missile  Frigate  acquisition  plan  already  discus- 
sed, a  key  element  of  the  plan  was  to  select  a  "lead"  an6 
a  "secondary"  contractor  early  in  the  design  effort.   The 
function  of  the  "lead"  shipbuilder  was  to  assist  in  the 
"in-house"  design  effort  and  ultimately  to  build  the  first 
ship  under  a  "cost  plus"  type  contract.   The  purpose  of  this 
was  to  introduce  producability  into  the  design,  to  promote 

design  fam4.1iarlty  and  acceptance  of  performance  responsibil- 

60 
ity  by  the  contractor,  and  to  reduce  development  time. 

The  "secondary"  shipbuilder  was  involved  to  prevent  the 

introduction  of  producability  bias  by  the  "lead"  shipbuilder 

which  would  result  in  unfair  advantage  when  bidding  on  the 

follow  ship  contracts  and  to  provide  a  "fallback"  position 

in  case  lead  ship  contract  negotiations  turned  sour.  When  the 

first  increment  follow  ships  fixed  price  contract  bids  were 

received  in  late  summer  1975>  the  Navy  was  dismayed  to 
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receive  bids  from  only  two  contractors  -  the  designated 
"lead"  and  "secondary"  contractors.   Moreover,  the  bid  prices 
were  well  in  excess  of  the  "Design-to-Cost"  constraint,  -" 
reportedly  as  much  as  6o^  over.   It  can  be  speculated  that 
the  lack  of  participation  in  the  bidding  by  other  shipbulld  - 
ers  and  the  high  bids  submitted  were  due  to  one  or  more  of 
the  following  reasons 

a)  shipyard  loading  by  other  (primarily  merchant 
ship)  contracts, 

b)  the  poor  profits  and  losses  experience  on  previous 
contracts, 

c)  a  reluctance  to  accept  the  required  engagement  by 
the  government  in  the  contractor's  procedures  and  operations, 

d)  fear  that  the  escalation  provided  for  in  the  con- 
tracts for  inflation  will  be  insufficient,  as  It  has  been  In 
the  past, 

and, 

e)  the  Navy's  cost  estimate  was  far  too  low. 
Whether  or  not  the  current  difficulties  in  the  Guided 

Missile  Frigate  program  are  a  result  of  past  project  diffi- 
culties or  an  indication  of  the  failures  of  the  new  reforms 
is  indeterminable  at  the  present  time.   How  many  Guided 
Missile  Frigates  (50  were  originally  planned)  will  ultimately 
be  built,  whether  or  not  contractor's  will  accept  the  desired 
increased  engagement  by  the  government  in  "their  affairs". 
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pnd  the  future  of  constrained  ship  design  are  all  open  to 
conjecture.   The  Navy's  only  other  "Design-to-Cost"  ship 
class  to  date,  the  DG  Aegis,  was  cancelled  and  replaced  by 
the  nuclear  powered  Strike  Cruiser  (CSGN).   Is  is  an  "un- 
constrained design  and  one  not  indicative  of  the  'Design- 
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to-Cost'  concept." 

Chapter  II  will  consider  the  case  for  a  comprehensive 
and  structured  ship  evaluation  system. 
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CHAPTER  T  CONCLUSIONS  AND  SUMMARY 

Three  distinguishable  periods  of  naval  ship  acquisition 
policy  have  occurred  since  WorDd  War  II.   The  periods  are 
referred  to  as  the  "Conventional",  "Concept  Formulation/ 
Contract  Definition"  or  "Total  Package  Procurement",  and 

'Current". 

The  aprroaches  have  differed  in  the  degree  of  centrali- 
zation of  control  within  the  Department  of  Defense;  contrac- 
tor -  government  relationships;  the  degree  of  systems  analy- 
sis; whether  the  design  was  accomplished  by  the  government, 
private  industry,  or  by  both;  and  the  types  and  numbers 
of  control  systems. 

The  objectives  have  remained  basically  the  same:  to 
deliver  the  ships  needed  for  the  nation's  defense,  when 
they  are  needed,  and  at  an  affordable,  promised  price. 
The  visible  results  have  also  been  similar  -  disappointing, 
to  say  the  least. 
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CHAPTER  IT 
THE  CASE  FOR  A  NAVAL  SHIP  EVALUATION  SYSTEM 

INTRODUCTION 

This  chapter  con«'lders  the  major  factors,  s^ipporting 
pptaMlshment  ,'^f  a  comprehensive  ship  evaluation  system. 
Thes'^  factors  include  the  historically  short  lived  acqui- 
sition policies  (relative  to  project  duration),  over-emphasis 
on  short  term  ^oal  achievement,  and  the  need  for  a  ship 
acquisition  life-cycle  historical  record  and  data  bank. 

It  is  concluded  that  successful  establishment  of  a 
comprehensive  evaluation  system  will  result  in  a  better 
basis  than  now  exists  for  formulating  high  potential  acqui- 
sition system  improvements  and  some  desirable  increase  in 
emphasis  on  longer  term  goal  achievement.   A  danger  in  the 
evaluation  system  will  be  the  tendency  to  use  its  results 
for  personnel  performance  appraisal. 

t   PET'TER  BASIS  FOR  ACQUISITION  SYSTEM  CHANGES  IS  NEEDED 

As  discussed  in  Chapter  I,  defense  system  acquisition 
policies  and  strategies  have  been  unstable  relative  to  the 
considerable  length  of  time  required  to  develop  and  construct 
a  new  class  of  ships  (often  ten  to  twelve  years).   Three 
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major  departures  in  acquisition  policy  occured  in  less  time 
than  it  took  to  develop  and  procure  the  USS  NIMITZ,  the  first 
In  a  new  class  of  nuclear  powered  aircraft  carriers.   The 
primary  emphasis  in  weapons  system  acquisition  policy  changed 
from  maximum  technical  performance  during  the  Conventional 
Period,  to  "cost  effectiveness"  over  the  life-cycle  during 
Concept  Formulation/Contract  Definition,  to  achieving 
acceptable  performance  for  the  minimum  acquisition  cost 
during  the  Current  Period.   Although  the  NIMITZ  may  partially 
reflect  parts  of  all  these  policies,  the  "die  was  cast"  early 
in  its  development,  under  the  Conventional  Period  policies. 

The  Navy  is  paying  a  large  price  for  the  adaptability 
and  ease  of  conversion  (to  minimize  life-cycle  costs)  design- 
ed into  the  new  SPRUANCE  class  destroyers  (DD-963  class) 
under  construction  by  Litton.   This  was  a  keystone  of  the 
Concept  Formulation/Contract  Definition  strategy.   The  new 
Guided  Missile  Frigate  class,  however,  is  a  "tight"  design 
with  small  margins  (reservations  for  future  systems  weight 
and   volume  requirements).   This  is  a  direct  result  of  the 
acquisition  cost  and  displacement  constraints  placed  on  the 
design  resulting  from  the  Current  Period  policies. 

It  Is  easy  to  forget  when  judging  a  ship  which  policies 
were  in  effect  when  it  was  designed  and  constructed  (and 
also  what  the  acquisition  environment  was  like).   Without  a 
formalized,  consistent  evaluation  system  which  collects  auid 
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assesses  data  on  the  actual  results  of  the  acquisitions 
throughout  the  ships'  life-cycles,  the  Navy  will  never  know 
the  degree  of  successes  and  failures  realized  under  the 
various  acquisition  policies  and  strategies.   Without  this 
knowledge,  changes  to  the  acquisition  system  are  reactionary 
and  often  little  more  than  "shots  in  the  dark". 

INCREASED  EMPHASIS  ON  LONG  TERM  GOAL  ACHIEVEMENT  IS  NEEDED 

Short  term  goals  in  ship  acquisition  include  those 
concerned  with  acquisition  cost  and  delivery  schedule. 
Longer  term  goals  include  minimizing  total  life  cycle  costs 
and  providing  capability  in  the  design  to  adapt  successfully 
to  new  and  unforseen  mission  requirements  and  to  counter 
future  threats.   Both  types  of  goals  are  important,  depending 
on  the  requirements  of  the  need.   The  realities  of  the 
political/economic  environment  strongly  influence  the 
selection  of  goals  and  their  attainment,  as  does  the  need 
requirements.   The  following  few  sections  will  focus  on 
the  reasons  for  dominant  emphasis  on  short  term  goals  and 
why  longer  term  goals  may  be  more  important. 

SHORT  TERM  GOAL  ATTAINMENT  PRESSURES 

There  are  a  number  of  reasons  why  short  term  goals 
achievement  receives  more  emphasis  in  ship  acquisition  than 
longer  term  goals  achievement.   Short  term  goals,  generally 
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attainable  during  the  acquisition  phase,  are  more  easily- 
quantified  and  measured.    The  short  tenure  (relative  to  the 
development  and  acquisition  time)  of  key  participants  increa- 
ses the  pressure  to  optimize  short  term  goals.   A  typical 
project  will  have  two  or  three  different  project  managers. 
They  are  likely  to  achieve  or  fail  promotion  based  on  the 
results  of  the  acquisition  foreseeable  during  their  tenure. 
One  Ship  Acquisition  Project  Manager  interviewed  by  the 
author  noted  that  "For  a  successful  project  in  this  building, 
bring  it  in  within  cost  and  on  schedule.   Unfortunately,  the 
fleet  will  not  see  it  that  way." 

By  "cost"  the  project  manager  meant  "acquisition  cost", 
and  "in  this  building"  refers  to  the  parent  organization  of 
the  project,  the  Naval  Sea  Systems  Command.   The  last  iine 
indicates  the  project  manager's  awareness  that  the  operators 
are  concerned  with  longer  term  goals.   A  senior  Naval  Material 
Command  (parent  organization  of  the  Naval  Sea  Systems  Command) 
executive  recently  prepared  "lessons  learned"  from  two  troub- 
led projects.   He  noted  that 

"Our  current  reward  system  does  not  encourage  special 
attention  to  that  portion  of  the  program  which  will 
occur  'down  the  pike'  on  a  subsequent  project  mana- 
ger's watch.   Our  reward  system  recognizes  near  term 
achievement.   With  the  increased  tour  lengths  of  pro- 
ject managers,  this  problem  has  been  !parti6.11y  alleviatjed." 
The   Concept  Formulation/Contract  Definition  period  policy 
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directive  required  minimum  tour  lengths  of  three  years  for 

1 
project  managers.   Recent  emphasis  has  been  towards  tours  of 

four  or  five  years  with  relief  occurring  at  major  milestones. 

The  tenures  of  other  key  procurement  participants  are 
normally  less  than  the  project's  life.   Key  officials  in  the 
Navy  and  the  Department  of  Defense,  military  or  civilian, 
rarely  are  in  office  more  than  three  or  four  years.   Few 
Congressmen,  particiilarly  Representatives,  can  afford  to 
empha^.ize  long  term  goal  achievement  at  the  expense  of  short 
term,  risible  results  as  n    rf^c^ult   of  their  frequent  reelec- 
tion rrquJrements.   Additionally,  Congress  does  not  presently 
have  the  capability  to  assess  the  long  term  results  of  acqui- 
sition programs.  Tar.    end  result  is  that  many  military  and 
public  officials  who  sponsor  and  approve  the  acquisition  of 

new  defense  systems  ?re  not  around  to  evaluate  the  program 

2 
results. 

Economic  Realities 


The  move  away  from  minimizing  life-cycle  costs  and  tow- 
ards tight  control  of  acquisition  costs  by  specifying  acqui- 
sition cost  constraints  as  design  param.eters  was  precipitated 
by 

a)  the  monujT.pntal  cost  overruns  of  the  1960's, 

b)  diminishing  real  resources  available  for  new 
acquisitions 
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3 
that 


and; 

c)  a  rapidly  aging  and  diminishing  fleet. 
A  recent  study  of  the  Navy's  acquisition  system  noted 

"Changing  national  priorities  have  reduced 
the  percentage  of  the  Federal  Budget  now 
devoted  to  defense  to  the  lowest  point 
since  1946." 
and, 

" the  U. S.  Navy  today  possesses  the 

lowest  force  level  It  has  known  since  before 
1950 " 


Reporting  and  Control  Systems  Effects 

4 
Anthony  observes  that  In  business, 

"Management  control  systems  tend  to  measure 
current  performance,  rather  than  long  term 
performance.   That  Is,  they  measure  the 
profit  earned  this  year,  rather  than  the 
effect  on  future  profits  of  actions  taken 
this  year." 
Hiere  Is  certainly  a  directly  analagous  situation  In  the 
defense  systems  acquisition  process.   The  Selected  Acquisi- 
tion Reports,  submitted  quarterly  to  Congress  by  major  weap- 
ons projects,  contain  a  lengthy  analysis  of  variances  from 
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acquisition  cost  estimates.   The  sections  on  technical  per- 
formance estimate  changes  Is  brief  and  there  is  no  provision 

5 
for  life  cycle  costs  in  the  reports. 


Ponding 

New  weapons  systems  are  funded  annually  by  Congress. 
The  Shipbuilding  and  Conversion,  Navy  (SCN)  account  carries 
a  five  year  obligation  authority  for  the  annual  appropria- 
tions.  This  account,  until  recently,  covered  new  ship  devel- 
opment and  production.   Recently,  funding  of  new  ship  devel- 
opment and  construction  of  the  lead  ship  of  a  class  has  been 
provided  by  the  Research,  Development,  Testing  and  Evaluation 
(R,D,T&E)  account.   This  budget  category  only  authorizes  a 
single  year  obligational  authority.   Long  range  planning  is 
extremely  difficult  under  such  a  short  funding  time  horizon. 
Erratic  congressional  action  on  the  yearly  appropriations  is 
not  uncommon,  and  this  compounds  the  problem  and  serves  to 
emphasize  short  term  goal  achievement. 

THE  IMPORTANCE  OF  LONG  TERM  GOALS 

The  threat  of  obsolescence  in  new  ships  at  the  time 
they  are  delivered  for  service  is  real.   The  missions  and 
threats  they  are  designed  to  cope  with  may  change  dramatically 
during  the  long  development  and  acquisition  period.   In  the 
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average  ship's  thirty  years  service  life,  missions,  threats 

and  available  systems  will  continue  to  change  as  a  result 

of  rapid  strides  in  technology.   Certainly  the  ultimately    ^ 

Important  test  of  ship  performance  is  how  it  performs  the 

missions  it  is  called  upon  to  perform,  whenever  that  might 

be. 

The  cost  to  develop  and  acquire  a  new  ship  is  typically 

only  about  one  fifth  of  the  total  expenditures  over  the  life 

6 
cycle  of  the  ship.   Over  emphasis  on  the  short  term  goal  of 

minimum  acquisition  cost  can  not  help  but  negatively  impact 

the  total  life  cycle  cost.   Increased  automation  in  a  ship 

results  in  smaller  manning  requirements,  but  the  automation 

increases  the  acquisition  cost.   Larger  manning  requirements, 

however,  significantly  impact  the  operating  costs  both 

through  increased  ship  ^Ize  (as  much  as  730  cubic  feet  per 

man)  and  the  dramatically  increasing  costs  of  wages  and 

benefits.   Another  example  concerns  the  long  life,  low  main* 

tenance  coatings  and  materials  becoming  available  which  are 

costly  to  purchase  but  may  favorably  impact  total  life  cycle 

cost.   Chief  of  Naval  Operations  Zumwalt  realized  the  severe 

impact  on  the  increasingly  limited  Navy  resources  of  the 

high  operating  and  maintenance  costs  of  the  aging  fleet.   He 

embarked  on  a  plan  to  eHmitiate  rmj^t^ly   many  of  the  costly 

old  ships  in  order  to  free  funds  for  new  ship  construction. 

In  conclusion,  ship  acquisition  long  term  goals  (e.g. 
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minimum  total  life  cycle  costs  and  capability  to  adapt 
rapidly  to  new  threats  and  missions)  are  important  but  the 
balance  of  pressures  on  the  acquisition  process  are  towards 
short  term  goal  achievement.   A  consistent  and  comprehensive 
evaluation  system,  operable  throughout  the  ship's  life  cycle, 
can  help  shift  this  balance  by  evaluating  and  making  visible 
the  degree  of  attainment  of  long  term  goals. 

A  COMPLETE  HISTORICAL  RECORD  IS  HEEDED 

The   recent  effort  to  improve  cost  estimating  within 
the  Department  of  Defense  (see  Chapter  I)  by  the  use  of 
comparative  parametric  cost  estimates  has  been  hampered  by 
the  lack  of  historical  return  costs  of  previous  weapons 
acquisitions.   Engineering  cost  estimates  and  actual  design 
control  are  developed  along  the  lines  of  a  work  breakdown 
structure  in  accordance  with  Military  Standard  88l,  "Work 
Breakdown  Structures  for  Defense  Material  Items". 

However,  the  accrual  accounting  system  established 
July  1,  197^ f  for  the  procurement  appropriations  uses  a 
different  cost  category  breakdown  which  does  not  correlate 
easily  with  the  work  breakdown  structures  used  by  the  pro- 
jects.  A  recent  effort  to  develop  a  crosb  referencing  sys- 
tem between  the  design  and  engineering  cost  estimating  work 
breakdown  structures  and  the  accounting  system  breakdown 
failed.   An  additional  difficulty  in  building  a  return  cost 
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historical  data  base  is  the  previously  discussed  contractor 
reluctance  to  divulge  actual  costs. 

The  current  increased  emphasis  on  traceability  of  per- 
formance «  cost  and  schedule  estimates  was  noted  in  Chapter  I. 
Historical  records  of  the  project  are  required  to  be  main- 
tained by  the  project  staff  but  the  records  may  or  may  not 
be  available  to  new  projects  and  are  of  varying  formats 
and  quality. 

After  the  acquisition  phase,  a  lot  of  cost  and  perform- 
ance data  is  generated  in  different  accounts  and  by  different 
systems.   There  are  at  least  a  half  dozen  different  cost 
accounts  for  the  Navy  which  contain  parts  of  the  life  cycle 
costs  of  ships.   Life  cycle  costs  such  as  those  of  mainten- 
ance, training  and  replenishment  facilities  and  operations 
are  not  always  identified  with  the  ship  classes  receiving 
the  support. 

PSRFORNAlfCE  DATA 

Currently  during  acquisition,  the  performance  of  a  ship 
system  is  being  demonstrated  more  rigorously  than  ever 
before.   Actual  hardware  demonstrations  are  replacing  paper 
estimates  earlier  in  the  process  in  many  cases.   At  the  end 
of  the  acquisition  phase,  complete  ship  trials  are  conducted 
by  the  Board  of  Inspection  and  Survey  to  assess  the  perform- 
ance of  the  ships. 
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After  acquisition,  there  are  numerous  sources  of  inform- 
ation about  actual  ship  performance.   In  recent  years,  actual 
combat  performance  has  been  available.   In  peacetime,  ship's 
undergo  a  constant  series  of  inspections  of  variou*  types. 
They  participate  in  fleet  training  exercises,  undergo  ship 
"refresher  training",  and  compete  with  other  ships  in  their 
operating  groups  for  coveted  "Efficiency"  awards  which  are 
based  on  the  outcome  of  a  series  of  standard  exercises.   The 
Board  of  Inspection  and  Survey  and  the  Propulsion  Examining 
Boards  conduct  periodic  comprehensive  inspections  of  material 
and  operating  readiness.   Information  about  the  reliability, 
maintainabiliy  and  performance  of  equipments  is  generated 
from  the  "3M"  system  of  scheduled  preventive  ^laintenaaee  and 
corrective  maintenance  reporting.   Operational  ships  report 
the  status  of  their  capabilities  to  perform  mission  require- 
ments through  the  Joint  Chiefs  of  Staff  Force  Status  and 
Identity  Reports.   Significant  degradations  due  to  equipment 

amd  system  casualties  or  personnel  deficiencies  are  reported 

9 
using  the  "Naval  Combat  Readiness  Criteria". 

Prom  the  above  and  other  sources,  pertinent  data  f  trs 
about  the  performance  of  the  ships  and  their  costs  are 
generated.   To  the  author's  knowledge  however,  there  is  no 
concerted  effort  to  collect  and  assimilate  all  the  pertinent 
data  and  utilize  it  to  evaluate  the  long  term  worth  of  the 
ships.   Such  a  purpose  would  be  served  by  a  central  evalua- 
tion system. 

en 


THE  EyALUATION  SYSTEM  AND  PERFORMANCE  APPRAISAL 

The  evaluation  system  is  not  being  proposed  as  a  control 
system.   Control  systemsare  short  horizon  oriented  and  there 
are  sufficient  and  entirely  adequate  systems  (see  Chapter  I) 
in  existence  to  help  manage  the  acquisition  process  and  eval- 
uate the  cost  effectiveness  of  goal  achievement  during  the 

10 
acquisition  phase. 

There  are  several  reasons  why  the  evaluation  system  to 
be  outlined  in  the  next  chapter  should  not  be  used  fir  personnel 
performance  appraisal.   Personnel  appraisal  demands  a  degree 
of  fairness  and  objectivity  that  will  not  be  attainable  in  a 
system  that  is  designed  to  evaluate  the  long  term  results  of 
a  process  as  complex  and  long  lived  as  a  ship  acquisition. 
It  may  be  true  that  the  responsibility  of  management  is  to 
be  fair  to  subordinates,  not  necessarily  to  design  a  person- 
nel appraisal  system  that  is  completely  fair  and  objective 

11  12 

in  itself.    However,  as  noted  by  Christensen,  et  al, 

"By  far  the  most  important  problem  of  measure- 
ment is  that  increased  interest  in  the  measure- 
ment of  performance  against  standards  brings 
increased  danger  that  the  executive  evaluation 
program  may  encourage  performance  which  detracts 
from  rather  than  supports  the  overall  strategy." 
Tliis  tendency  results  in  actions  which  are  directed 


65 


towards  optimizing  the  performance  system  results  at  the 

expense  of  the  projects  goals ^  particularly  long  term  goals. 

The  more  quantitative  and  simplistic  the  evaluation  criteria 

the  greater  is  the  tendency.   A  common  example  which  arises 

in  business  is  the  evaluation  of  profit  center  managers  on 

simple  return  on  investment.   The  reluctance  of  an  executive 

so  evaluated  to  invest  in  new,  more  efficient  but  higher 

book  value  production  equipment  is  understandable  but  is 

of  ten  detrimental  to  the  long  term  profit  performance  of  the 

firm. 

13 
Lo range  discusses  the  problem  this  way 

"There  is  a  danger  that  the  performance  tracking 

process  will  lead  to  decision-making  behavior 

which  violates  the  overall  organizational  goal 

congruence  requirement.   This  is  partly  due  to 

the  "technical"  measurement  problems  

These  often  stem  from  a  desire  to  capture 

complex  and  multi face ted  phenomena  by  means 

of  a  few  variables " 

The  true  function  of  a  measurement  system  is  to  get 

an  idea  of  the  problems  limiting  achievement  rather  than  to 

14 
place  blame.     No  single  measure  or  measurement  system 

can  totally  assess  the  individuals  contribution  to  intermed- 
iate and  long  term  results  or  to  the  efforts  of  others.   This 
is  particularly  true  when  the  time  horizon  of  long  term  goals 
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1b  extremely  long  as  in  the  case  of  ship  acquisitions.   As 
an  avowed  purpose  of  the  evaluation  system  is  to  increase 
the  emphasis  on  long  term  goal  achievement,  any  temptation 
to  utilize  the  system  for  personnel  appraisal  should  be 
resisted. 

CHAPTER  II  CONCLUSIONS 

Unless  improvement  in  the  ship  acquisition  process  is 
to  be  left  to  chance,  the  degree  of  attainment  of  project 
goals  must  be  observed  and  measured.   Currently,  by  far  the 
most  emphasis  is  on  attainment  of  short  term  goals  e.g. 
minimizing  acquisition  cost.   The  most  important  goals  in 
ship  acquisition  may  be  longer  term  in  nature,  e.g.  minimi- 
zation of  total  life  cycle  costs  and  performance  adaptabil- 
ity. 

A  consistent  and  comprehensive  evaluation  system  which 
assesses  the  degree  of  success  of  ship  acquisitions  should 
be  developed.   A  key  function  of  the  system  will  be  to  col- 
lect and  integrate  the  various  data  concerning  life  cycle 
costs  and  performance  as  the  data  becomes  available  for  the 
different  ship  classes.   The  results  contained  in  this 
information  will  be  evaluated  against  the  goals  of  the  pro- 
ject, long  and   short  term.   The  resulting  assessments  will 
be  used  for  formulating  improvements  to  the  acquisition 
process.   The  historical  data  collected  should  also  be 
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of  use  for  budgeting,  improving  estimating  relationships 
and  providing  "lessons  learned"  to  new  acquisitions. 

The  evaluation  system  should  not  be  used  for  the  apprai- 
sal of  personnel  performance  nor  is  it  intended  for  acquisi- 
tion phase  project  control. 

Chapter  III  will  consider  the  development  of  a  naval 
ship  evaluation  system,  present  individual  sets  of  goals 
criteria  for  the  three  major  goal  areas,  and  discuss  two 
approaches  to  the  overall  assessment  of  naval  ship  success. 
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CHAPTER  III 
A  SHIP  SUCCESS  EVALUATION  SYSTEM 

INTRODUCTION 

In  this  chapter  a  framework  for  a  ship  evaluation  system 
is  outlined.   In  the  first  part  of  the  chapter,  a  discussion 
of  the  desirablp  traits  of  a  ship  evaluation  system  is  presen- 
ted.  The  prerequisites  of  the  evaluation  system  include 
establishment  of  the  need,  a  set  of  prioritized  goals  based 
on  the  need,  and  a  definition  of  success  in  naval  ship  acqui- 
sition.  A  preferred  definition  of  success  is  offered. 

The  second  part  of  Chapter  III  includes  development  of 
sets  of  goal  criteria  for  each  of  the  three  major  goal  areas. 
This  is  followed  by  outlines  of  a  simple  manual  model  and 
a  more  sophisticated,  computerized  simulation  model  for  com- 
bining the  sets  of  criteria  and  arriving  at  an  overall  evalua- 
tion of  success.   The  latter  model  requires  greater  resources 
but  may  result  in  a  more  valid  and  objective  evaluation 
scheme. 

It  is  concluded  that  the  proposed  evaluation  system 
is  feasible,  can  be  designed  so  that  the  criteria  are  relevant 
and  the  assessments  arrived  at,  though  not  completely  objec- 
tive, will  be  more  objective  and  meaningful  than  any  currently 
available. 
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DESIRABLE  SYSTEM  TRAITS 


The  desirable  traits  of  a  ship  acquisition  evaluation 

system  are  much  the  same  as  those  required  of  accounting 

1 
principles  -  relevancy,  objectivity  and  feasibility.    Rele- 
vant in  that  the  measure  used  is  meaningful  and  useful, 
objective  to  the  extent  that  the  assessment  criteria  is  not 
influenced  by  personal  biases  or  Judgments  and  feasible 
meaning  that  the  costs  of  implementation  and  the  complexity 
of  the  system  are  within  reason. 

Total  objectivity  requires  purely  quantitative  analyses. 
This  ideal  is  unobtainable  when  evaluating  a  system  as  com- 
plex as  a  ship  if  relevancy  and  feasibility  are  to  be  main- 
tained.  The  performance  of  a  naval  ship  is  particularly  dif- 
ficult to  quantify  completely.   However,  as  the  stated  purpose 
of  the  evaluation  system  is  to  determine  success  as  an  aid  in 
developing  improvements  in  the  acquisition  process  rather  than 
to  serve  as  a  control  system  per  se,  the  necessary  sacrifice 
of  objectivity  is  not  so  serious.   The  goal  will  be  to  utili- 
ze single  and  combinative  quantitative  criteria  as  long  as  it 
makes  sense  to  do  so;  at  the  same  time  recognizing  that  employ- 
ment of  an  "irrelevant  criterion  simply  because  it  lends  it- 

2 
self  to  quantification  is  a  poor  exchange  far  alleged  otjjectivltyi' 

EVALUATION  SYSTEM  PREREQUISITES 

It  is  assumed  *hat  a  legitimate  and  well  defined  military 

need  has  been  established  and  a  ship  has  been  selected  as  the 
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best  alternative  system  to  satisfy  that  need.^  Although  the 
proposed  evaluation  system  has  been  specifically  excluded 
from  serving  as  a  conventional  control  system,  the  basic 
steps  required  to  formulate  it  are  similar:  goal  specifica- 
tion, establishment  of  goal  attainment  criteria,  results  data 
collection,  and  evaluation  of  the  degree  of  goal  attainment. 

GOAL  SPECIFICATION 

The  first  requirement  is  that  a  set  of  goals  for  the 
weapons  system  acquisition  must  be  clearly  specified.   These 
goals  are  derived  directly  from  the  need  and  should  reflect 
the  planned  resources  allocation  to  meet  the  need. 

Not  only  must  the  goals  be  clearly  specified;  they  must 
be  prioritized,  or  weighted.   The  reason  for  this  requirement 
is  that  major  project  goals  (cost,  schedule  and  performance) 
are  Interdependent.   A  schedule  change  for  example,  will  inev- 
itably result  in  a  cost  change,  particularly  in  an  inflation- 
ary economy.   The  generally  inverse  relationship  between  acq- 
uisition cost  and  total  life  cycle  costs  was  noted  in  Chapter 
II,   Permanent  performance  deficiencies  may  result  in  a 
remaining  gap  in  the  need  which  must  be  filled  ultimately. 

It  is  impossible  to  optimize  simultaneously  interdepen- 
dent goals.   Some  directives  and  project  plans,  however,  seem 
to  require  this.   For  example,  a  recent  Chief  of  Naval  Opera- 
tions directive  notes  that  there  had  been  little  documentation 
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available  " of  the  effort  expended  to  maximize  ships' 

mission  effectiveness  while  at  the  same  time  minimizing 

ship  costs. "   (emphasis  added) 

For  the  most  part,  however,  current  policy  directives 

refer  to  a  need  to  achieve  a  proper  balance  between  major 

goals.   For  example,  Secretary  of  the  Navy  Instruction  5000.1 

"System  Acquisition  in  the  Department  of  the  Navy",  specifies 

"Continual  tradeoffs  shall  be  performed  to  optimize 

the  balance  between  Initial  acquisition  cost, 

estimated  life  cycle  cost,  schedule,  and  operational 

capability."  (emphasis  added) 

Ttie  proper  balance,  i.e.  the  priority  of  goals,  as  well 

as  actual  selection  of  the  goals,  should  be  tailored  to  each 

project  as  the  requirements  of  the  needs  and  the  priorities 

of  the  needs  are  different.   Further,  the  relative  importance 

of  the  goals  should  be  specified  prellminay  to  the  search  for 

the  "best"  system  to  fill  the  need.   As  noted  in  the  1972 

5 
"Report  of  the  Commission  on  Government  Procurement", 

"The  need  should  be  separated  from  any  particular 

system,  and  goals  should  be  defined  independently 

of  the  performance,  cost  and  schedule  characteristics 

of  any  particular  system the  exploration  of 

alternative  solutions  requires  that  these  things 

be  specified. 

♦  A  statement  of  the  problem  to  be 
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solved  or  the  deficiency  in  mlseion  capa- 
bility, including  those  conditions  that 
have  created  the  problem  or  deficiency. 

*  '^^   goft^s  to  be  achieved  by  the  acqui- 
sition program  and  their  relative  import- 
ance, including  level  of  mission  capability, 
program  cost  and  when  the  capability  should 
be  available. 

♦The  boundary  conditions  that  Bust  be  met 
by  any  system,  including  constraints  on  phy- 
sical size,  operating  conditions,  tactics, 
and  the  talents  of  the  users." 
(emphasis  added) 

Once  a  specific  ship  acquisition  project  is  formed  in 
response  to  the  need,  the  prioritized  goal  set  should  be 
formally  specified  in  the  project's  charter.   This  will 
provide  the  acquisition  group  with  the  specific  guidance 
needed  to  conduct  consistent  tradeoff  analyses. 

DEFINITION  OP  SUCCESS 

Different  ship  acquisition  participants  and  obervors 
view  success  differently.   As  previously  noted,  some  empha- 
size achievement  of  one  or  more  short  term  goals,  others 
are  interested  in  only  the  long  term  worth  of  the  acquisitioa 
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Assuming  that  a  prioritized  goal  set  is  established 

which  accurately  reflects  the  requirements  of  a  legitimate 

need,  the  only  definition  of  success  in  ship  acquisition 

which  maKes  sense  is  how  well  the  actual  results  of  the 

acquisition  match  the  goals,  including  their  relative 

6 
importance.   The  goal  set  must  be  flexible  to  the  point 

that  it  continues  to  reflect  the  requirements  of  a  changing 

need. 

CRITERIA  CONSIDERATIONS 

The  selection  of  appropriate  standards  by  which  to 
measure  actual  results  against  is  not  always  clearcut.   The 
choice  is  influenced  by  the  goal  in  question,  the  phase  of 
the  acquisition,  the  observability  and  measurability  of  the 
data,  and  whether  the  assessment  concerns  actual  versus 
planned  results  or  one  ship  class  against  another. 

Estimates  for  the  major  goal  areas  arc  required  in  the 
Operational  Requirement  Docximent  which  is  prepared  prelimi- 
nary to  Conceptual  Design.  These  estimates  are  changed  and 
refined  as  the  design  progresses.  The  estimates  become 
thresholds  when  written  into  a  Secretary  of  Defense  approved 
Decision  Coordinating  Paper  after  the  first  Defense  Systems 
Acquisition  Review  Council  presentation  (refer  to  Figure  1-2 
in  Chapter  I). 
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Most  Important  to  meaningful  evaluation  Is  that  changes 
to  estimates  are  fully  substantiated  as  to  cause  and  are 
completely  documented.   Traceability  of  estimates  permit  the 
evaluation  results  arrived  at  by  using  one  standard  to  be 
adjusted  for  a  prior  or  later  standard  as  desired. 

No  one  type  of  criteria  Is  feasible  for  the  evaluation 
of  all  the  different  aspects  of  a  ship  acquisition.   The 
question  of  quantified  versus  qualitative  criteria  was  addres- 
sed previously.   Performance  measures  for  complex  systems  are 
generally  probabilistic  at  best,  usually  in  the  form  of  plots, 
The  necessary  combination  of  such  criteria  to  arrive  at 
higher  level  systems  and  functions  evaluation  results  in  an 
increasing  need  for  Judgment. 

Measurement  of  results  in  the  cost  area  can  be  mostly 
quantitative  and  in  the  form  of  variaince  analysis  (meaning 
comparison  of  actual  results  to  planned  results  rather  than 
the  statistical  definition).   The  major  questions  to  be  an- 
swered are  what  are  the  appropriate  costs  to  evaluate  and 
what  are  the  appropriate  baselines  to  use.   Still,  Judgment 
of  the  economic  environment  in  which  the  acquisition  took 
place  is  necessary  in  order  to  determine  controllable  and 
uncontrollable  effects  on  costs. 

The  assessment  of  success  in  the  schedule  area  deals 
primarily  with  how  closely  the  ships  are  delivered  to  the 
schedule  determined  from  the  need.  Quantitative  variance 
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analysis  is  useful  here  also.   The  choice  of  what  is  the 
appropriate  "required  schedule"  may  not  be  elearcut. 

After  a  short  discussion  of  data  requirements  for  the 
evaluation  system,  individual  sets  of  criteria  for  each  of 
the  major  goal  areas  will  be  developed  in  detail.   Then, 
two  different  approaches  to  the  combination  of  those  criiieria, 
so  that  an  overall  evaluation  of  success  can  be  arrived  at, 
will  be  developed. 

DATA  REQUIREMENTS 

The  evaluation  system  should  have  the  data  available 
that  is  needed  to  describe  accurately  the  acquisition  and 
permit  in-depth  assessment  of  what  its  results  were  and  why. 
What  must  be  avoided  is  the  collection  of  data  which  is  dupli- 
cative, irrelevant  or  more  costly  to  collect  than  its  value. 

It  is  difficult  to  imagine  that  new  data  systems  will  be 
required  to  implement  the  evaluation  system  due  to  the  numer- 
ous data  sources  discussed  previously.  The  new  function  pro- 
posed is  primarily  that  of  collecting  and  assimilating  data.. 

Close  adherence  to  DOD  Directive  5000.19,  "Policies  for 
the  Management  and  Control  of  DOD  Information  Requirements", 
will  help  ensure  data  requirements  are  economical  and  concise, 
This  directive  directs  that  new  information  requirements  be 
screened  for  suitability,  and  that  they  be  designed  to  meet 
only  essential  needs. 
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SHIP  PERFORMANCE  EVALUATION 

The  performance  of  the  ship  system  is  key  when  conslder- 

7 
ing  the  relative  worth  of  ships.    An  acquisition  may  be 

delivered  on  schedule  and  meet  its  cost  target,  but  if  it 

fails  to  perform  the  missions  required  of  it,  in  no  sense 

can  it  be  considered  successful.   Conversely,  in  the  opinion 

of  at  least  one  Ship  Acquisition  Project  Manager  interviewed 

by  the  author,  costs  and  schedule  overruns  do  not  remain 

visible  for  long  after  delivery  of  the  ships  and  thus  the 

performance  of  the  units  is  the  only  goal  area  which  counts 

in  the  long  run.   The  weakness  of  this  viewpoint  lies  in  the 

insidious  effect  acquisitions  which  are  out  of  control  can 

have  on  other  acquisition  projects,  and  thus  on  the  overall 

defense  posture.   Neither  short  term  nor  long  term  goals  can 

be  ignored. 

SHIP  PERFORMANCE  DEFINED 

The  definition  of  ship  performance  must  include  any 
feature  that  impacts  directly  or  indirectly  on  the  ability 
of  a  ship  to  carry  out  its  assigned  mission.   This  broad 
definition  was  earlier  offered  by  Graham.   Graham  includes 
combat  ability  (first  and  most  important),  mobility,  surviva- 
bility, habitability,  maintainability,  and  future  capability 
within  the  scope  of  the  definition.   Eckhart  points  out  that 
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any  measure  of  ship  effectiveness  must  include  availability  - 

9 
"ships  at  sea  versus  ships  in  being".    Sanders  reiterated 

10 
the  point  as  follows 

"Too  many  of  our  ships  are  only  partially 
ready  to  carry  out  their  missions  on  a 
continual  basis.   This  places  our  Fleet 

and  Force  commanders,  ,  into  the 

position  of  assigning  tasks  on  the  basis 
of  which  ship  is  ready  rather  than  which 
ship  has  the  most  built-in  capability  to 
do  the  Job." 
Depending  on  how  the  performance  features  are  defined, 
others  may  be  listed,  such  as  the  detectability  of  the  ship. 
Since  a  change  in  capability  in  one  feature  area  inevitably 
impacts  on  another,  the  evaluation  of  overall  ship  perform- 
ance must  consider  many  features,  some  of  which  are  difficult 
to  quantify. 

For  example,  a  new  requirement  to  increase  the  size  of 
the  ship's  crew  will  increase,  unless  compensated  for  by  re- 
ductions in  other  performance  features,  the  displacement 
and  volume  of  the  ship  (possibly  as  much  as  5  tons  and  750 
cubic  feet  per  man).   The  increased  displacement  and  size 
will  slow  the  ship  down  and  reduce  its  fuel  endurance.   To 
reclaim  this  loss  in  mobility  performance  will  require 
increased  horsepower  in  the  propulsion  plant.   This  in  turn 
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will  further  decrease  the  endurance  and  additional  fuel 
tankage  will  have  to  be  added.   The  increased  size  and  weight 
of  the  propulsion  plant  and  fuel  tankage  increases  the  dis- 
placement and  volume  of  the  ship,  etc.   It  goes  without 
saying  that  each  necessary  increase  in  the  ship  will  add 
to  the  acquisition  and  other  life  cycle  costs. 

Why  it  is  necessary  to  include  all  performance  features 
when  considering  ship  performance  is  made  clearer  to  the 
design  layman  when  an  analysis  of  their  impact  is  made  in 

terms  of  "pure"  performance,  such  as  the  number  of  guns, 

11 
as  Graham  demonstrated.    Pbr  example,  it  was  found  that  the 

increased  space  built  into  our  ship  machinery  spaces  (increa- 
sed maintainability)  designed  since  World  War  II  is  equiva- 
lent in  impact  to  two  5"  light  weight  guns  on  a  representa- 
tive destroyer.   Thus  the  question,  simply  put,  is  "Is  the 
'tradeoff  of  larger  specific  machinery  box  with  armament, 

representing  a  compromise  of  maintainability  and  hab- 

itability  with  combat  ability,  a  good  one?"  The  answer  to 
this  question  rests  in  how  well  the  overall  results  match;::" 
the  project  goals  set,  i.e.,  the  degree  of  success. 

A  FRAMEWORK  FOR  PERFORMANCE  EVALUATION 

Objective  evaluation  of  complex  system  performance 
achievement  has  eluded  systems  analysts  and  controllers.   As 


81 


12 
Baumgartner  observed  in  I963, 

"The  main  question  to  be  resolved  at  this  point 
In  time  is  how  to  measure  performance  and  how 
to  plan  performance.   Effective  answers  will 
be  developed  Jupt  as  surely  as  the  final 
performance  objectives  of  a  project  are 
achieved  by  effort  and  direction  rather 
than  by  accident." 
It  appears  that  Baumgartner  was  overly  optimistic  as  no 
completely  objective  and  comprehensive  measurement  systems 
have  been  developed,  to  this  writer's  knowledge,  that  can 
account  for  different  inputs,  acquisition  environments,  and 
output  form  and  purpose. 

However,  the  development  during  the  past  three  years  of 
the  Top  Level  Requirements  (TLR)  and  Top  Level  Specifications 
(TIjS)  process  provides  the  basic  framework  needed  for  a  con- 
sistent and  comprehensive  (if  not  completely  objective)  ship 
performance  evaluation  system.   The  key,  as  will  be  discussed 
later,  will  be  to  extend  this  new  requirements  setting  and 
performance  measure  specification  process  beyond  the  acquisi- 
tion phase. 

The  following  sections  will  discuss  in  detail  the  Top 
Level  Requirements/Top  Level  Specifications  process  and  the 
contents  of  the  documents.  A  third  key  element  of  the  per- 
formance evaluation  system  to  be  developed  is  a  system  of 

82 


consistent  procedures  for  calculating  the  performance  meas- 
ures in  the  Top  Level  Specifications.   An  initial  effort 
in  this  direction  exists  and  will  be  discussed.   The  existing 
and  planned  standardized  procedures  (termed  Performance  Anal- 
ysis Data  Sheets)  are  presented,  as  are  the  Top  Level  Speci- 
fications performance  measures,  in  a  composite  table  of 
performance  features  (from  the  Top  Level  Requirements)  and 
criteria. 

The  Top  Level  Requirements/Top  Level  Specifications  Process 

As  noted  in  Chapter  I,  a  draft  Top  Level  Requirements, 

based  on  the  results  of  Conceptual  Design  tradeoff  studies 

and  cost  effectiveness  analyses,  is  issued  at  the  beginning 

13 
of  Preliminary  Design.    Tlie  performance  requirements  in 

excess  of  minimum  essential  requirements  are  then  traded  off 

against  cost  and  schedule  requirements. 

The  Top  Level  Specifications  breaks  down  the  performance 

statements  in  the  Top  Level  Requirements  into  basic  func- 

14 
tions.    An  operational  capability  to  "search,  detect,  clas- 
sify and  track  air  targets",  for  example,  is  divided  into  the 
four  basic  functions  listed.   A  performance  measure  is  used 
for  each  of  the  four.   The  function  "search/detect"  is  logi- 
cally measured  by  range.   The  Top  Level  Requirements  may 
specify  a  minimum  essential  value  or  an  acceptable  range  of 
range  values.   The  Top  Level  Specification,  however,  will 

83 


state  the  predicted  range  performance  .-^5 

A  further  example  is  that  the  basic  operational  capabil- 
ities in  mobility  include  "speed,  endurance,  seakeeping  qual- 
ities, repair  capabilities,  etc."   For  an  aircraft  carrier 
escort,  the  Top  Level  Requirements  may  specify  a  minimum  es- 
sential speed  in  a  given  sea  condition  but  the  Top  Level  Spec- 
ifications will  answer  with  predicted  speed  performance  and 

16 
speed  degradation  in  given  sea  conditions. 

As  the  Top  Level  Specifications  "presents  the  achievable 
ship  within  the  cost  constraint,  it  may  well  be  that  some 

performance  measures  in  (the)  Top  Level  Requirements  cannot 

17 
be  met  ..."   A  Change  in  the  cost  constraint,  required  per- 
formance or  design  practices  must  then  be  negotiated  and  docu- 
mented in  the  Top  Level  Requirements  and  Top  Level  ^)eclfLMitiona 

At  the  end  of  Preliminary  Design,  "final"  Top  Level  Re- 
quirements and  Top  Level  Specifications  (containing  design 
specification  and  performance  measures)  are  issued.   Subse- 
quently, all  changes  to  either  document  are  serialized  and 
fully  documented.  (See  Figure  1-2.) 

18 
Top  Level  Requirements  Content 

The  major  sections  of  the  Top  Level  Requirements  include 
Operational  Capabilities,  Planned  Use  and  Cost  Constraints. 
The  document  is  a  translation  of  the  military  need  into  speci- 
fic requirements.   A  brief  description  of  it  contents  follows. 
A.  Operational  Capabilities 

1.  Warfare  Areas.  This  is  a  description  of  the 
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"combat  tasks  and  functions  the  ship  is  intended  to  perform 
in  appropriate  mission  areas".   These  mission  areas,  detailed 
in  thr  Chief  of  Naval  Operations  Instruction  3301.2, "Naval 
Combat  Readiness  Criteria",  include  surface  warfare,  anti-air 
warfare,  anti-submarine  warfare,  etc.  Ttie    "warfare  areas" 
correspond  directly  to  Graham's  "combat  ability"  performance 
feature.  The   section  includes  how  the  ship  relies  on  and 
complements  the  capabilities  of  other  types  of  units  in  the 
face  of  particular  threats.  Requirements  related  to  the  de- 
tection of  the  ship  by  different  sensor  types  and  requirements 
related  to  the  survival  of  the  ship,  e.g.  "hardness"  to  shock 
and  air  blast,  are  also  described  in  this  section. 

2'  Mobility  Area.   This  includes  the  "operational  cap- 
abilities required  in  the  movement  and  transit  of  the  ship". 
Listed  in  performance  terms  are  speed,  endurance,  seakeeping, 
maneuvering, replenishment  constraints  and  other  operational 
constraints.   Draft  and  beam  restriction  (  to  allow  harbor 
and  canal  passage)  as  well  as  damage  control  and  engineering 
casualty  control  requirements  are  described. 

?•   Corranand  and  Contro.^  Ares.  This  area  describes  the 
provisions  for  planning  and  controlling  operations  of  the  ship 
and  associated  forces  pursuant  to  the  assigned  missions.   Tlie 
extent  of  the  operational  capabilities  and  interfaces  with 
other  systems  (by  digital  computer  on  large  modern  combatants) 
is  detailed. 

4.  Fleet  Support  Operations  Area.  These  are  the  "opera- 
tional capabilities  required  of  the  ship  to  provide  aid  and 


support  to  other  units." 

^*  Non-Coirbat  Operations  Area.  These  are  the  opera- 
tional capabilities  pxpected  which  are  not  directly  related 
to  coir.bat."   Included  are  search  and  rescue,  etc. 
B.  Planned  Use 

^*  finvi T'onmpnt .  This  area  includes  a  description  of 
the  ship's  expected  operating  environment  as  well  as  its 
allow??ble  effect  on  the  environment.   The  standard  environ- 
mental conditions  against  which  ship  and  subsystem  perform- 
ance are  to  be  predicted  as  well  as  pollution  related  matters 
are  detailed  in  this  section. 

-•  Operational  Availability  and  Ship  Utilization. 
This  section  "describes  the  ship  availability  and  readiness 
to  get  underway  for  specific  ship  readiness  conditions  in 
normal  and  altered  conditions." 

3.  Maintenance.  Overhaul ,  and  Supply  Support  Con- 
cepts.  This  section  is  the  intended  maintenance  and  support 
philosophy  for  the  ship.   Minor  and  major  repair  frequency 
and  duration  and  the  level  of  on  board  maintenance  capability 
and  repair  parts  availability  are  specified. 

^'  Manning.   The  "manning  philosophy,  together  with 
operational  Items  which  affect  manning,  such  as  degree  of 
system  automation,  watch  requirements,  and  experience  levels 
and  such  other  personnel  related  matters  which  constrain  ship 
manning"  are  detailed  here.   Habitability  and  any  special 
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safety  requirements  are  included. 

5.  Wartime  Use.   Itiis  section  "describes  the  anti- 
cipated operational  profile  and  maintenance  cycles  during 
times  of  hostilities  when  the  ship  is  subject  to  participa- 
tion in  combat".   Subtopics  include  speed  and  time  profile, 
maintenance  and  overhaul  cycle,  and  operating  and  logistic 
support. 

6.  Peacetime  Use.   The  same  information  as  that 
specified  in  the  above  section  is  presented,  but  for  during 
peacetime. 

7.  Other.   In  this  area  are  planned  use  items 
which  do  not  clearly  fall  under  the  previous  six  topics. 
Applicable  exceptions  to  existing  regulations  and  directives, 
as  well  as  pertinent  directives  which  are  not  normally  invok- 
ed, are  spelled  out. 

19 
Top  Level  Specifications 

The  Top  Level  Specifications,  the  design  answer  to  the 
required  operational  capabilities  and  planned  use  assumptions 
of  the  Top  Level  Requirements,  presents  measures  of  ship 
performance  and  overall  ship  attributes  utilizing  three  dif- 
ferent classification  systems.   Tliese  systems  are  measures  of 
overall  ship  attributes  achieved,  measures  of  ship  system 
functional  performance  achieved  (traces  directly  to  the  Top 
Level  Requirements  Operational  Capabilities)  and  measures  of 
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performance  for  ship  subsystems. 

The  ship  system  functional  performance  measures  included 
in  the  Top  Level  Specifications  are  generally  the  most  mean- 
ingful ones  to  the  ship  operator.   This  serves  the  purpose  of 
the  evaluation  system  well  as  a  primary  interest  is  in  the 
ship's  functional  capabilities  at  the  mission  performance 
level.   The  performance  in  any  one  mission  area,  e.g.  anti- 
submarine warfare,  may  then  be  assessed  by  combining  evalua- 
tions of  the  relevant  functions,  e.g.  search,  detect,  clas- 
sify, track  and  engage,  of  the  anti-submarine  warfare  con- 
tributing systems. 

In  some  cases,  traceability  exists  between  the  Top  Level 
Requirements  and  Top  Level  Specifications  performance  meas- 
ures through  common  attribute  names,  e.g.  speed,  seakeeping, 
stability,  vulnerability,  availability  and  noise.   There  also 
exists  direct  traceability  between  the  Top  Level  Requirements 
Habitability  Standards  and  the  Top  Level  Specifications  Habi- 
tability  Summary,  the  Top  Level  Requirements  Supply  Support 
Concepts  and  the  Top  Level  Specifications  Supportability, 
etc. 

Performance  Analysis  Data  Sheets 

The  Naval  Ship  Engineering  Center,  design  arm  of  the 
Naval  Sea  Systems  Command,  is  currently  developing  a  system 
to  provide  "the  analytical  procedures  required  to  calculate 
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the  performance  measures  in  the  Top  Level  Specifications  for 

20 
a  ship  design.     The  purpose  of  these  "Performance  Analysis 

Data  Sheets"  is  to  "set  forth  a  single  method  for  consistent 
application  which  will  precisely  define  the  Top  Level  Speci- 
fications performance  measures  and  make  visible  all  assump- 
tions, margins,  loss  allowances,  safety  factors,  etc.,  used 

21 
in  its  calculation".    Performance  Analysis  Data  Sheets 

standardized  procedures  and  criteria  are  not  being  developed 
where  it  is  considered  that  adequate  standardized  guidance 
already  exists. 

Performance  Measures  Set 

By  combining  the  performance  requirements  and  con- 
straints in  the  Top  Level  Requirements,  the  performance  pre- 
dictions and  measures  in  the  Top  Level  Specifications,  and 
by  utilizing  the  existing  and  planned  standardized  criteria 
calculations  procedures  specified  in  the  Performance  Analysis 
Data  Sheets,  a  set  of  criteria  for  evaluating  a  ship's  per- 
formance throughout  its  life  cycle  can  be  developed.   Meas- 
urement of  "success"  in  the  attainment  of  performance  goals 
can  then  be  accomplished  by  utilizing  primarily  existing 
and  planned  systems.   As  noted  previously,  the  major  change 
required  will  be  to  collect  actual  results  data  and  assess 
the  performance  against  the  requirements  throughout  the  life 
cycle  rather  than  just  during  the  acquisition  phase. 
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The  table  in  Appendix  A  contains  the  proposed  set  of 
performance  features  evaluation  criteria  developed  as  discus- 
sed above.   Included  in  the  table  are 

a)  the  name  and  a  description  of  the  performance 
measures  and  ship  attributes  considered  useful  to  the  evalua- 
tion system,  generally  derived  from  the  Top  Level  Specifica- 
tions, 

b)  the  Top  Level  Requirements  performance  objectives 
and  planned  use  areas  which  are  directly  measured  by  the 
performance  measure  (indicated  by  a  "T"  for  direct  traceabil- 
ity)  or  are  at  least  partially  evaluated  by  the  performance 
measure  (indicated  by  an  "I"  for  influence), 

and, 

c)  the  standardized  procedures  for  calculating  the 
performance  measures  which  have  been  developed,  or  are  plan- 
ned for  development.   Note  that  only  the  title  or  a  very 
brief  description  of  the  Performance  Analysis  Data  Sheets 

or  other  standardized  procedure  is  listed.   Where  a  procedure 
is  considered  inadequate  by  the  author  for  the  purposes  of 
the  evaluation  system,  or  is  not  known  to  be  planned  for 
development,  a  recommendation  is  included.   Such  recommenda- 
tions are  enclosed  in  parenthesis. 

The   next  two  sections  will  develop  criteria  of  costs  and 
schedule  goals.   Then  the  composite  evaluation  of  success 
will  be  addressed. 
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COST  EVALUATION 

Acquisition  cost  is  the  most  visible  and  controver- 
sial factor  in  naval  ship  acquisition.   This  situation  re- 
sults from  the  monumental  cost  overruns  of  the  past  two 
decades  and  the  huge  amounts  required  to  purchase  new  major 
weapons  systems.   A  single  new  nuclear  powered  aircraft 
carrier  costs  in  excess  of  one  billion  dollars  to  acquire. 

Never-the-less,  a  case  was  made  in  the  previous  chapter 
that  the  total  life  cycle  cost  is  the  more  appropriate  cost 
target  to  minimize.   However,  selection  of  the  appropriate 
cost  goals  and  their  relative  iraporteuice  for  each  project  is 
left  to  the  goal  setting  process.   The  purpose  of  the  eval- 
uation system  is  to  measure  how  closely  the  results  match 
the  original  or  current  aet  of  prioritized  goals. 

ACQUISITION  PHASE  COST  CRITERIA 

There  is  no  scarcity  of  cost  information  during  the 
acquisition  phase.   There  is  a  problem,  as  previously  noted, 
of  obtaining  contractor  return  costs.   However,  as  the  de- 
gree of  engagement  with  the  contractors  increases  and  more 
of  them  meet  the  Cost  and  Schedule  Control  Systems  Criteria 
(see  Chapter  I),  useful  return  costs  information  should  be- 
come more  available  and  be  of  greater  validity. 

The  Selected  Acquisition  Reports  (SARS)  required  by 
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?2 
Congress  concentrate  on  acquisition  costs  and  contain 

detailed  .-acquisition  cost  variance  analyses.   Acquisition 

cost  baselines  for  the  evaluation  system  could  be  chosen 

from  the  cost  estimates  reported  in  the  SARS,  the  Design- 

to-Cost  constraints,  or  the  estimates  in  the  Decision 

Coordinating  Papers.   These  estimates  should  all  be  in 

basic  agreement. 

The  Naval  Sea  Systems  Command's  "Design-to-Cost  Guide 

for  new  ships"   stipulates  that  major  programs  will  use 

SAR  specified  guidelines  (variance  analyses)  for  explaining 

cost  growth.   The  same  directive  requires  that 

"A  historical  record  of  cost  constraints, 

cost  estimates,  actual  costs  as  th^ become 

available,  and  cost  influencing  factors 

will  be  maintained  in  order  to  provide  the 

basis  for  explaining  any  cost  increases. " 

The  Design-to-Cost  constraints  are  formulated  in  terms 
24 
of  "unit  sail-sway  costs".    This  is  defined  as 

" the  estimated  unit  cost  of  construc- 
tion of  follow-on  ships  as  delivered  including 
government  furnished  equipment,  and  the  instal- 
lation pnd  adaption  of  computer  programs  but 
excluding  non-recurring  costs  and  outfitting 
cost,  in  constant  dollars  of  a  given  fiscal 
year,  based  on   a  given  number  of  ships  in  the 
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series  and  a  given  delivery  schedule." 

Program  acquisition  cost  variances,  as  reported  in  the 

25 
quarterly  SARS,  are  classified  as  resulting  from 

a)  quantity  change 

b)  engineering  change 

c)  support  item  change 

d)  schedule  change 

e)  unpredictable  change  (strikes,  acts  of  God) 

f)  economic  change 

g)  estimating  change  (due  to  errors  or  revisions 
in  estimating  methodology  and  relationships) 

h)  contract  performance  incentives 

i)  contract  cost  overruns  (underruns) 
Iliese  reports  therefore  contain  an  adequate  selection  of 
acquisition  cost  criteria  and  analyses. 

POST  ACQUISITION  PHASE  COST  CRITERIA 

As  previously  noted.  Department  of  Defense  Directive 

5000,1,  "Acquisition  of  Major  Defense  Systems"  requires 

that  "cost  parameters  shall  be  established  which  consider 

the  cost  of  acquisition  and  ownership"  (emphasis  added). 

The  Office  of  the  Secretary  of  Defense  Cost  Analysis  Improve- 

26 
ment  Group  charter   specifies  that  the  group  is  responsible 

for  providing  a  review  and  evaluation  of  independent  and 

program  cost  estimates  submitted  at  the  Defense  Systems 
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Acquisition  Review  Council  presentations.   The  charter  fur- 
ther stipulates  that 

"These  cost  reviews  shall  consider  all  elements 
of  system  costs,  including  procurement,  opera- 
tions and  support  as  appropriate."  (emphasis  added) 

The  "Department  of  the  Navy  Cost  Analysis  Program"  was 

27 
announced  in  March,  1975.   The  establishing  directive   re- 
quires that  costs  be  considered 

" for  all  phases  of  the  weapon  system,  in- 
cluding the  conceptual,  definition,  acquisition 
and  operational . " 
The  program  requires  the  Chief  of  Naval  Operations  to 

a)  "Ensure  that  realistic  cost  estimates  are 
provided  for  the  planning,  programming,  and  budgeting  of 
systems  and  equipment  acquisition.", 

b)  'bevelop  and  maintain  a  cost  data  base  for  initial 
and  follow  on  cost  estimating,  cost  review  and  validation, 
and  budgeting. ", 

c)  "Provide  guidance  and  develop  costing  methodology 
to  ensure  that  the  total  costs  of  acquisition  and  ownership 
of  weapon  systems  and  force  units  are  available  to  decision 
makers." 

and, 

d)  "Maintain  force  costing  models  to  facilitate 
consideration  of  alternative  force  structures  and  marginal 
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changes."  (emphasis  added) 

Integration  of  the  various  sources  and  categories  of 
life  cycle  costs  is  required  for  application  of  the  evalua- 
tion system.   The  above  plans  and  requirements,  if  carried 
through,  should  result  in  that  integration. 

Most  of  the  data  will  be  quantitative,  and  thus  the 
criteria  can  be  quantitative.   However,  some  Judgments  must 
be  interjected  as  to  what  indirect  costs  will  be  charged 
to  a  particular  ship  class.   These  overhead  charges  include 
the  cost  of  maintaining  and  operating  support  facilities 
such  as  shipyards,  repair  ships,  replenishment  ships,  sup- 
ply depots,  etc.   Also,  qualitative  analysis  of  the  effects 
of  the  resource  constraints,  and  the  political,  economic 
and  procurement  environments  is  necessary,  as  previously 
noted. 

Table  III-l  provides  a  summary  of  the  suggested  base- 
line estimates  for  the  principal  life  cycle  costs  categoriei 
Actual  costs  incurred  should  be  available  initially  from, 
the  Selected  Acquisition  Reports  submitted  quarterly  by 
the  projects,  and  later  from  the  Chief  of  Naval  Operations 
maintained  force  costing  models  and  data  bank.   Schedule 
goal  criteria  will  be  addressed  in  the  next  section. 
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SCHEDULE  EVALUATION 

A  major  consideration,  along  with  low  unit  acquisition 

cost  for  the  Guided  Missile  Frigate  program  was  to  make 

contract  award  for  production  of  the  lead  ship  in  less  than 

28 
two  years  after  the  requirement  was  levied  in  1971.    Two 

and  one  half  to  three  years  is  usually  required  for  this 

process.   The  design  management  plan  of  having  the  Naval 

Ship  Engineering  Center  immediately  start  Contract  Design 

upon  completion  of  Preliminary  Design  and  soon  afterwards 

bring  in  the  lead  ship  and  secondary  contractors  to  aid  in 

the  design  was  developed  primarily  due  to  the  schedule 

urgency. 

The  early  ballistic  missile  programs  were  counted  by 

most  observors  as  successes  rather  than  failures  even  though 

they  Incurred  large  cost  overruns  and  there  were  deficien*- 

cies  in  their  initial  operational  performance.   They  were 

"successful"  because  the  goals  during  the  "missile  gap"  era 

put  primary  emphasis  on  the  early  development  of  some  sort 

29 
of  deterrent  capability.    A  JikKD   Corporation  study  of  the 

ICBM  programs characterized  them  as  an  example  "where  an  im- 
mediate demonstration  force  is  needal  whether  or  not  it 

30 
works. " 

Again,  the  point  to  be  made  is  that  success  should  be 
measured  against  the  requirements  of  the  need  as  expressed 
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in  a  prioritized  goal  set.   It  is  entirely  possible  that,  at 
least  initially,  rapid  introduction  of  the  system  into  the 
operating  forces  will  be  the  dominant  requirement. 

As  is  the  case  for  acquisiticMO  costs,  the  delivery  sched- 
ule of  a  new  ship  class  is  monitored  in  the  Decision  Coordi- 
nating Paper/Defense  Systems  Acquisition  Review  Council 
and  the  Selected  Acquisition  Report  processes.   Delivery 
schedules  are  often  unstable  as  a  result  of  one  or  more  of 
the  below  reasons, 

a)  overly  optimistic  estimates  of  development  and 
construction  time, 

b)  cost  increases  which  force  reductions  and  delays 
in  awarding  production  contracts, 

and, 

c)  yearly  fluctuations  in  the  number  of  ships  author- 
ized by  Congress. 

The  actual  delivery  schedule  is  known  well  before  actual 
life  cycle  costs  and  mission  performance  results.   The  recom- 
mended criteria  is  a  simple  variance  analysis,  as  required 
by  the  Selected  Acquisition  Reports,  using  the  baseline 
estimate  as  contained  in  the  Decision  Coordinating  Paper. 
Schedule  variances  resulting  from  the  causes  cited  above 
will  require  qualitative  analyses  of  their  effects. 

In  the  remainder  of  this  chapter,  the  combination  of 
criteria  into  an  overall  success  evaluation  system  is  acBressed. 
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OVERALL  EVALUATION 

Having  selected  approptiate  individual  criteria  for 
features  of  each  of  the  three  major  goals,  it  remains  to 
assess  the  combined  results  of  the  measure.   One  of  the 
prerequisites  specified  was  that  clearly  defined,  priori- 
tized goals  would  be  stated  for  a  project  as  derived  from 
the  established  need.   This  can  be  done  by  simply  ranking 
the  goals  in  order  of  priority  or  by  assigning  weighting 
factors  to  the  goals. 

S<ich  a  specification  of  goal  priorities  can  be  viewed 
as  the  utility  function,  or  criterion,  to  be  optimized  in 
the  new  ship  class.   As  the  goal  weighting  is  determined 
from  the  requirements  of  the  need,  it  may  be  necessary  to 
change  the  respective  weightings  as  the  ship  class  progres- 
ses through  its  life  cycle.   A  key  element  of  success  of  a 
given  project  will  be  flexibility  enough  in  the  design  to 
adapt  to  new  mission  requirements  and  resource  constraints. 
Comprehensive  documentation  and  tracking  of  the  goal  speci- 
fications will  permit  the  evaluators  to  ii^tigef  inrtell,egbntiy 
the  degree  of  success  relative  to  the  current  or  past  goal 
sets,  or  utility  functions. 
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INDEPENDENT  EVALUATION  OF  MAJOR  GOALS 

A  strong  case  has  repeatedly  been  made  against  evalua- 
ting one  or  the  other  major  goals  singaJt'ly.  llk>  ypettormsxai^ 
level  is  worth  any  cost,  and  likewise,  inadequate  perform- 
ance to  accomplish  the  mission  requirements  is  no  bargain. 
The  requirements  of  the  need  and  the  relative  priority  of 
the  need  to  other  defense  needs  must  be  considered.   Only 
by  evaluating  total  resxxlts  against  the  prioritized  goal 
set  can  actual  success  be  Judged. 

This   is  not  to  say,  however,  that  the  comprehensive 
data  collected  on  ship  acquisitions  should  not  be  utilized 
for  comparing  results  in  any  particular  area  or  for  improv- 
ing single  design  procedures  and  historically  based  estima- 
ting relationships.   Such  uses  should  be  eo«e  of  the  major 
benefits  of  the  evaluation  system. 

OVERALL  EVALUATION  MODELS 

Two  approaches  to  integrating  the  criteria  into  an 
overall  evaluation  model  will  now  be  discussed.   The  approa- 
ches are  basically  the  same,  but  differ  considerably  in  their 
sophistication  and  degree  of  automation. 
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A  Manual  Evaluation  Model 


Existing  and  already  planned  systems  have  been  utili- 
zed to  a  great  extent  in  formulating  individual  goal  feature 
criteria  and  for  data  sources  for  the  evaluation  system.   It 
is,  in  fact,  the  emergence  of  these  new  systems  which  have 
made  comprehensive  evaluation  throughout  a  ship's  life  cycle 
possible. 

In  order  to  combine  the  interdependent  criteria  and 
goals,  some  sort  of  model  must  be  utilized.   The  first  ap- 
proach to  constructing  and  using  an  evaluation  model  contin- 
ues the  low  additional  resource  requirement  theme.   In  ad- 
dition to  low  resource  requirements,  a  simple  approach 
which  relies  heavily  on  the  Judgment  of  the  evaluators 
recognizee  the  following, 

a)  the  extreme  complexities  of  the  system  being  eval- 
uated, 

b)  the  intricate  interdependencies  of  cost,  perform- 
ance and  schedule  goals, 

c)  the  difficulties  of  finding  relevant  quantified 
measures  for  every  goal  area, 

d)  the  uniqueness  of  every  project  which  will  require 
a  different  evaluation  model  in  each  case, 

and, 

e)  the  problem  of  reducing  every  measure  to  a  common 

dimension  to  permit  straightforward  combination. 
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These  problems  raise  serious  doubts  about  the  validity  and 
usefulness  of  a  computerized  model  which  would  basically  only 
require  data  input  and  then,  with  little  evaluator  maftlpula- 
tion,  turn  out  an  overall  assessment  of  success. 

A  simpler,  manual  scheme  will  probably  produce  as  good 
or  better  results  at  lower  cost.   The  qualitative  Judgments 
necessary  to  make  the  results  meaningful  negate  the  value 
of  a  computer  aid.   Additionally,  the  assessments  will  be 
more  acceptable  to  acquisition  process  participants,  due 
to  the  relatively  simpler  approach  used  to  arrive  at  them. 
So  go  the  arguments. 

The  Basic  Scheme 
In  each  goal  area,  the  myriad  of  data  inputs  would  be 
collected  and  assessed  using  the  individual  criteria  settled 
upon  at  the  inception  of  the  project.  The  evaluation  criteria 
should  have  assigned  to  them  distributed  weights  indicating 
the  relative  importance  of  each  subsystem  or  subarea  being 
measured.   These  criteria  and  the  W6ighting  of  them  should 
be  developed  by  the  acquisition  project  and  approved  through 
the  Decision  Coordinating  Paper/Defense  Systems  Acquisition 
Review  Council  process.   The  criteria  presented  in  the  pre- 
vious sections  should  be  useful. 

This  is  not  a  completely  new  requirement  for  the  pro- 
jects.  Department  of  Defense  Instruction  7041.3,  "Economic 
Analysis  and  Program  Evaluation  for  Resource  Management", 
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requires  that, 

"Criteria  by  which  effectiveness  can  be  evaluated 
are  to  be  clearly  specified  at  the  inception  of 
a  program/project." 
Development  of  the  criteria  set  and  assignment  of  relative 
importance  weightings  to  the  criteria  by  the  project  should 
promote  acceptance  of  the  evaluation  results. 

A  number  of  schemes,  including  weighted  point  assign- 
ments, weighted  variances  or  simply  rankings,  could  then  be 
devised  which  would  serve  to  inte{;;rate  the  weighted  criteria 
within  each  goal  area.   This  is  the  same  type  process  used 
by  many  projects  to  evaluate  contractor  proposals  submitted 
in  response  to  bid  requests.    Finally,  by  applying  the 
weights  contained  in  the  utility  function  (prioritized  goal 
set)  to  the  results  in  each  of  the  goal  areas,  an  evaluation 
of  the  overall  success  of  the  project  is  possible. 

A  great  deal  of  judgment  will  be  required  in  assigning 
the  weights  and  applying  the  criteria.   The  evaluation 
results  will  have  to  be  qualified  in  detail.   The  subjective 
nature  of  the  assessment,  though  a  "number"  is  arrived  at, 
should  never  be  lost  ight  of.   The  final  resulting  evalua- 
tion, never-the-less,  should  be  far  superior  to  the  frag- 
mented, biased  judgments  emanating  from  various  acquisition 
observors  and  participants. 


104 


A  Brief  Example 

Table  III-2  presents  a  sampling  of  example  quantita- 
tive analyses  for  a  fictitious  ship  class.   The  methods  of 
calculating  the  goa]  features  variances  are  arbitrary  and 
other  methods  may  be  preferable. 

After  completion  of  the  individual  analyses,  the  next 
step  is  to  assess  the  quantitative  variajices  together  with 
all  relevant  subjective  information.  The   variances  may  be 
adjusted  (reasoning  should  be  carefully  documented)  or  sim- 
ply accompanied  by  modifying  qualitative  analyses. 

The  aggregation  of  variances  to  higher  levels  of  goal 
features  can  then  be  accomplished  by  using  pre-determined 
weightings.   For  example,  consider  the  Costs  goals.   Table 
III-3  indicates  how  the  evaluators  might  calculate  weighted 
variances  which  indicate  the  degrees  of  attainment  of  the 
Costs  goals.   The  sum  of  the  weighted  variances  is  then 
an  indication  of  the  degree  of  attainment  of  the  prioritized 
Costs  goals. 

Table  III-4  is  an  example  of  how  a  final  figure  which 
indicates  the  degree  of  success  of  the  acquisition  (i.e.  how 
well  the  results  match  the  prioritized  goals  set)  can  be 
arrived  at.   In  the  example,  a  negative  ^undesirable)  overall 
variance  of  .02  from  the  prioritized  goal  set  is  calculated. 

This  example  exercise  is  only  an  indication  of  the 
simple  quantitative  techniques  which  might  be  of  use  to  the 
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Table  III-3 
EXAMPLE  HOSTS  GOAL  ATTAIN^!ENT  CALCULATION 
Cost  Category        Weighting 
■'.Pi-ogram  Acquisition 


Varlancp 

Weighted 
Variance 

(.17) 

(.05) 

.06 

.01 

Cost  .3 

P.  Logistic  Support/ 
Ac'riltional  Procurement    .1 
Costs 

^.  Operatlng/Suppcrt 

Costs  A  .02  .01 

4.  Conversion  Costs       .?  (.07)         (.01) 

Overall  Costs  Weighted  Variance   (.04) 

Table  TTI-4 

EXAMPLE  OVERALL  GOAL  ATTAINMENT  CALCITLATION 

GopJ         Weighting      Variance  Weighted  Variance 

Cost           .3            (.04)  (.01) 

Performance     .5             .00  .00 

Schedule        .2            (.03)  (.01) 

Overall  Weighted  Variance   (.02) 
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evaluators.  The  admonitions  in  the  previous  section  against 
over-emphasizing  or  misusing  the  results  of  such  an  analysis 
merit  constant  awareness. 

31 
A  Sophisticated  Evaluation  Model 

Advocacy  of  a  more  sophisticated  evaluation  model  for 
ship  acquisition  success  evaluation  centers  around  the  same 
observations  of  the  complexities  involved  as  made  in  support 
of  the  manual  model.   However,  there  is  evidence  that  non- 
quantifiable  goals  and  goal  attainment  criteria  are  often 
largely  ignored  when  evaluation  is  done  on  a  primarily  intui- 
tive basis.   Goals  criteria  which  can  not  be  easily  quanti- 
fied should  be  evaluated  by  knowledgeable  experts  and  assign- 
ed quantified,  though  subjective,  ratings.   Truly  non-quanti- 
fiable goal  criteria  will  serve  little  useful  purpose  as  part 
of  pertinent  criteria  used  to  determine  the  degree  of  success 
attained  by  a  ship  acquisition  project. 

Also,  the  limitation  of  human  mentality  severely  limits 
the  number  of  goal  interdependencies  that  can  be  analysed 
under  the  previous  model.   The  decreasing  costs  of  time 
sharing  and  batch  process  computer  systems,  the  steady 
advancement  of  new  systems  analysis  techniques,  and  the 
previously  noted  continued  expansion  of  defense  acquisition 
information  and  control  systems  (creation,  processing  and 
preservation  of  large  amounts  of  quantitative  and  qualitative 
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data),  all  suggest  the  feasibility  of  a  sophisticated 

computerized  simulation  model. 

Beginning  in  the  early  1960's,  the  Department  of  Defense 

has  made  extensive  use  of  systems  analysis  computer  models 

to  aid  in  appraising  alternative  weapon  systems  and  to 

integrate  long  term  planning  with  yearly  budgeting.   This 

proposal  amounts  to  an  extension  of  the  usefulness  of  the 

long  range  planning  models  to  include  follow  up  evaluation 

of  actual  results  fit  to  the  requirements. 

That  the  final  assessment  of  succes?  will  be  done  by 

the  model,  independent  of  Judgment,  is  not  advocated.   As 

Hitch  stated  in  I967, 

"Systems  analysis  is  simply  a  method  to  get 

before  the  decision  maker  the  relevant  data, 

organized  in  the  way  most  useful  to  him.   It 

is  no  substitute  for  sound  and  experienced 

military  judgment,  and  it  is  but  one  of  the 

many  kinds  of  information  needed  by  the 

decision  maker. " 

Model  Construction 

The  first  step  in  building  the  model  is  the  same  as 

that  for  the  simpler,  manual  model.   The  set  of  prioritized 

goals  (the  utility  function)  for  the  project  is  specified. 

This   is  the  success  reference  against  which  the  overall 

results  will  be  assessed. 

Early  in  the  project  life,  management  scientists  will 
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conduct  a  series  of  Interviews  with  key  design,  financial 
and  management  personnel  associated  with  the  project.  The 
purpose  of  their  interviews  will  be  to  discover  the  inter- 
dependencies  of  the  project  goals,  subsystem  and  subgoal 
relationships,  and  the  predicted  range  of  effects  on  project 
outcome  from  externalities  such  as  the  funding  schedule  and 
amounts,  the  procurement  and  econonic  environments,  etc. 

The  relationships  will  be  designed  into  the  simulation 
model.   The  selected  criteria  will  be  weighted  and  entered. 
Performance  measures  represented  by  probabilistic  plots  will 
be  modeled  by  mathematical  functions.   Constraints  which 
describe  cost  ceilings  and  minimum  performance  requirements 
will  be  used  to  flag  "unacceptable"  results. 

The  net  contribution  that  subgoal s  attainment  will 
make  towards  each  of  the  multiple  goals  must  be  completely 
structured  in  the  model.   This  can  be  accomplished  by  first 
specifying  separate  probability  distributions  for  every 
underlying  factor  and  then  specifying  factor  interdependen- 
cies  to  form  Joint  probability  distributions. 

The  individual  simulations  of  each  of  the  major  goal 

areas  can  then  be  jointly  simulated  by  the  same  techniques 

as  above.   This  approach  is  along  the  same  lines  as  the 

prototype  intertemporal  goal  programming  model  outlined 

33 
by  Ijiri. 
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Model  Usage 

As  data  on  30t,u?!l  results  become  available,  they  will 
be  inputted  to  compare  with  the  original  estimated  probabil- 
ity distributions  and  point  estimates.   The  latter  can  be 
modified  to  reflect  actual  results  and  an  increasingly 
accurate  assessment  of  success  obtained. 

It  is  doubtful  that  any  ships  will  ever  encounter  the 
exact  operational  and  combat  scenarios  set  up  in  the  Top 
Level  Specifications.   However,  the  scenarios  can  be  modeled 
in  the  war  gaming  activities  of  the  Chief  of  Naval  Operations 
organization.   Actual  ship  performance  results  can  be  extrap- 
olated into  the  war  gaming  data  bank  as  they  become  available 
and  actual  scenario  performance  predicted  with  increasing 
accuracy.   These  war  gaming  results  can  then  be  compared 
to  the  baseline  distributions  in  the  evaluation  model. 

The  computer  model  results  in  combination  with  qualita- 
tive information  available  to  the  evaluators  should  enable 
a  relatively  straightforward  assessment  of  overall  project 
success.   By  studying  the  varying  degrees  of  success  of 
different  projects  in  conjunction  with  the  projects'  differ- 
ent imposed  acquisition  policies,  acquisition  environments 
and  management  and  design  techniques,  intellegent  formulation 
of  high  success  potential  improvements  to  the  acquisition 
system  process  and  structure  should  be  enhanced. 

The  construction  and  implementation  of  the  computer 

models  should  be  done  with  as  much  involvement  of  project 
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personnel  as  possible.   Obviously,  special  management  science 
skills  are  needed.   The  operations  research/systems  analy- 
sis personnel  utilized  should  have  project  experience  where 
possible,  and  be  primarily  management  oriented.   A  great 
deal  of  interaction  between  acquisition  participants  and 
the  model  builders  is  an  absolute  roust  if  the  model  is  to 
be  valid  and  considered  useful. 

CHAPTER  III  CONCLUSIONS  AND  SUMMARY 

A  ship  evaluation  system  should  be  objective^  relevant 
and  feasible.   Total  objectivity  is  prevented  by  the  complex 
interdependencles  and  multi-variables  involved  which  are 
often  difficult  to  quantify  independently,  much  less  when  in 
combination. 

Prerequisites  to  ship  success  evaluation  include  a 
legitimate  defense  need  which  is  best  served  by  a  naval 
ship,  a  prioritized  set  of  cost,  schedule  and  performance 
goals  derived  from  the  need  requirements,  and  definition 
of  what  success  is.   Success  is  best  determined  by  how  well 
actual  naval  ship  results,  throughout  the  ship  life  cycle, 
fit  the  prioritized  goal  set. 

The  final  steps  in  developing  the  evaluation  system  are 
criteria  selection,  evaluation  model  construction,  and 
provisions  for  necessary  drta  collection.   Criteria  and 
data  requirements  can,  for  the  most  part,  be  met  by  existing 
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and  already  planned  rystems. 

Two  approaches,  differing  in  resource  requirements, 
sophistication  and  degree  of  automation,  have  been  presented 
for  evaluating  "overall  success".   The  use  of  a  "manual 
model"  for  overall  evaluation,  it  is  argued,  "^  s  preferable 
because  validity  of  the  results  requires  a  great  deal  of 
judgment  be  interjected  into  the  system. 

The  computerized  simulation  program  model,  augmented 
by  the  evalua tors'  judgment,  can  account  for  more  of  the 
complex  interdependencies  involved  in  evaluating  ship 
success.   However,  it  requires  greater  resources  to  develop 
and  use  and  its  results  may  not  be  as  acceptable  to  acquisi- 
tion system  participants  and  observors. 

Neither  of  the  models  measure  up  to  complete  objectivity 
We  never  supposed  they  would.   The  manual  model  is  certainly 
feasible  and  could  probably  be  implemented  in  a  short  time. 
The  greatest  problem  would  probably  be  setting  up  the  col- 
lection of  the  myriad  of  life-cycle  data  in  a  central  loca- 
ton  and  processing  it  to  usable  form. 

The  computerized  model  is  less  feasible  in  that  it 
would  take  a  significantly  greater  amount  of  resources  to 
implement  and  manage.   It  Is  definitely  possible  over  a 
period  of  a  few  years,  however.   Ttie  relevancy  of  either 
model  is  dependent  on  the  skill  of  those  involved  in  the 
construction  and  implementation  of  *he  evaluation  system. 
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Although  the  work  required  and  Its  complexity  may  be 
apalling  to  some,  the  implementation  of  either  of  the  two 
approaches  could  result  In  great  payoffs  for  future  acquisi* 
tlon  efforts. 
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RECOMMENDATIONS 

Tt  Is  recommended  that  a  comprehensive  ship  life-cycle 
success  evaluation  system  be  developed  and  implemented. 
The  framework  provided  in  Chapter  III  for  such  a  system 
should  be  roughly  adhered  to.   Particularly  important  to 
follow  are  the  concepts  of  utilizing  existing  systems  out- 
put? and  insistence  on  clearly  specified  and  prioritized 
goals  at  the  inception  of  a  project. 

The  standardised  procedures  for  calculating  the  Top 
Level  Specifications  performance  measures  require  further 
development  and  expansion  to  presently  uncovered  areas. 
The  Performance  Analysis  Data  Sheets  provide  a  usable  frame- 
work for  this  effort. 

The  decision  on  whether  to  take  the  "manual"  model 
or  computerized  model  approach  for  development  of  the 
overall  success  evaluation  system  should  be  carefully 
we1,9:hed.   It  would  probably  be  prudent  to  start'  With- the 
lower  resource  requirement  system  and  then  transition  it 
towards  the  more  sophisticated*  computerized  model.   The 
analysis  of  this  decision  and  a  comprehensive  trial  imple- 
mentation of  a  ship  success  evaluation  system  is  recommended 
for  a  future  thesis  effort. 
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APPENDIX  A 


TABLE  OP  PERFORMANCE  MEASURES 
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KEY  TO  TABLE 

"TLS"       Top  Level  Specifications 
"TLR"       Top  Level  Requirements 
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